One-Touch Projector Alignment for 3D Stereo Display

ABSTRACT

A 3D image is created by using two digital projectors having overlapping projection regions. The two digital projectors both emit polarized light, and one are arranged at different angles to each other, such that individual images projected from respective digital projectors can be discerned by the use of polarized eyeglasses. Individualized light transport matrices are used to coordinate together the two digital projectors.

BACKGROUND

1. Field of the Invention

The present invention is related to methods of creating 3D image contentusing digital projectors, and particularly related to the creation of 3Dimaging content using projector-camera systems. The present invention isfurther related to methods of compensating for light diffusion and lightnoise in the compensating of projector-camera systems.

2. Description of the Related Art

When projectors and cameras are combined, hybrid devices and systemsthat are capable of both projecting and capturing light are born. Thisemerging class of imaging devices and systems are known in the researchcommunity as projector-camera systems. Typically, images captured by oneor more cameras, are used to estimate attributes about displayenvironments, such as the geometric shapes of projection surfaces. Theprojectors in these projector-camera systems then adapt their projectedimages so as to compensate for shape irregularities in the projectionsurfaces to improve the resultant imagery. In other words, by using acamera, a projector can “see” distortions in a projected image, and thenadjust its projected image so as to reduce the observed distortions.

In order to achieve this, the camera and projector need to be calibratedto each other's imaging parameters so as to assure that any observedimage distortion is due to irregularities in the projection environment(i.e. surface irregularities), and not due to distortions inherent tothe projector or camera, or due to their relative orientation to eachother.

Thus, a key problem that builders of projector-camera systems anddevices need to solve is the determination of the internal imagingparameters of each device (i.e. intrinsic parameters) and thedetermination of the geometric relationship between all projectors andcameras in the system (i.e. extrinsic parameters). This problem iscommonly referred to as that of calibrating the system.

Even after a system has been substantially calibrated, however, theissue of adjusting a projection to compensate for distortions in aprojected image is not straightforward. Identifying and compensating forprojection distortion can be a very computationally intensive operation,which has traditionally greatly limited its application tonon-specialized fields.

In an effort to better understand the calibration of projector-camerasystems, Applicants studied multi-camera imaging systems found in thefield of computer vision. Although such multi-camera imaging systemsconsist of only image photographing devices, and do not include anyimage projecting devices, a large body of work concerning thecalibration of such multi-camera imaging systems exists in the field ofcomputer vision, and it was thought that one might glean some insightfrom their approach toward calibrating multiple devices, albeit multipleimage photographing devices.

A commonly used method in computer vision techniques for calibrating acamera in an imaging system is described in article, “A flexible newtechnique for camera calibration”, IEEE Transactions on Pattern Analysisand Machine Intelligence, 22(11):1330-1334, 2000, by Zhengyou Zhang,which is herein incorporated in its entirety by reference. In thismethod, multiple images of a flat object marked with a number of knownfeature points (typically forming a grid) are captured by the camera,with the flat object posed at a variety of known angles relative to thecamera. The image location of each feature point is extracted, and sincethe relative location of each feature point is known, the collection offeature point locations can then be used to calibrate the camera. Whentwo or more cameras are present in an imaging system, the intrinsicparameters as well as the geometric relationship between the presentcameras can be estimated by having all cameras capture an image of theflat object at each pose angle.

Since projectors and cameras are very similar in terms of imaginggeometry, it might seem reasonable to postulate that techniques suitablefor calibrating cameras in multi-camera imaging systems might besuitable for calibrating projectors in projector-camera systems.However, since all camera calibration techniques require that the camerarequiring calibration (i.e. the imaging device being calibrated) capturea number of images, it would appear that camera calibration techniquescannot readily be applied to projectors since projectors cannot captureimages.

Therefore, in traditional projector-camera systems, at least two camerashave been needed, in addition to a projector. The two cameras arecalibrated first, typically using multi-camera imaging systemcalibration techniques, to establish a stereo camera pair. Morespecifically, these systems use a “bootstrapping” procedure to calibratethe two cameras and form the stereo camera pair. As it is known in theart, a stereo camera pair can be used to estimate depth (i.e. achieve apseudo perspective view) to establish a quasi-depth perception offeature points visible to the stereo camera pair. The calibrated stereocamera pair are then used to calibrate the projector. Basically, theestablishment of this quasi-depth perception is used to identify surfacedepth irregularities of a projection surface, and thereby of an imageprojected onto the projection surface. The projector can then becalibrated to compensate for the surface depth irregularities in theprojected image. In essence, to calibrate the projector using thisquasi-depth perception, the projector is first made to project featurepoints onto a display environment (i.e. the projection surface), whichmay have an irregular surface. The pre-calibrated, stereo camera pair isused to resolve the perspective depth location of the projected points.The projector can then be calibrated to compensate for surface/depthirregularities in the projection surface, as determined by the depthlocation of the projected points. While this bootstrapping technique isa tested-and-proven calibration method for projector-camera systems, itis not applicable to the calibration of self-contained projector-cameradevices, since it requires the use of pre-calibrated; strategicallylocated, external stereo camera pairs, and thus requires much operatorintervention.

Of related interest is a technique called dual photography proposed bySen et al. in article, “Dual Photography”, Proceedings ACM SIGGRRAPH,2005, which is herein incorporated by reference in its entirety. Dualphotography makes use of Helmholtz reciprocity to use images capturedwith real cameras to synthesize pseudo images (i.e. dual images) thatsimulate images “as seen” (or effectively “captured”) by projectors.That is, the pseudo image simulates a captured image as “viewed” by aprojector, and thus represents what a projector-captured image would beif a projector could capture images. This approach might permit aprojector to be treated as a pseudo camera, and thus might eliminatesome of the difficulties associated with the calibration of projectors.

Helmholtz reciprocity is based on the idea that the flow of light can beeffectively reversed without altering its light transport properties.Helmholtz reciprocity has been used in many computer graphicsapplications to reduce computational complexity. In computer graphicsliterature, this reciprocity is typically summarized by an equationdescribing the symmetry of the radiance transfer between incoming(ω_(i)) and outgoing (ω_(o)) directions as fr(ω_(i), ω_(o))=fr(ω_(o),ω_(i)), where fr represents the bidirectional reflectance distributionfunction (BRDF) of a surface.

Thus, dual photography ideally takes advantage of this dual nature (i.e.duality relationship) of a projected image and a captured image tosimulate one from the other. As is described in more detail below, dualphotography (and more precisely Helmholtz reciprocity) requires thecapturing of the light transport property between a camera and aprojector. More specifically, dual photography requires determination ofthe light transport property (i.e. light transport coefficient) relatingan emitted light ray to a captured light ray.

When dealing with a digital camera and a digital projector, however,dual photography requires capturing a separate light transportcoefficient relating each projector pixel (i.e. every emitted light ray)to each, and every, camera pixel (i.e. every light sensor that capturespart of the emitted light ray), at the resolution of both devices. Sincea digital projector and a digital camera can both have millions ofpixels each, the acquisition, storage, and manipulation of multitudes oflight transport coefficients can place real practical limitations on itsuse. Thus, although in theory dual photography would appear to offergreat benefits, in practice, dual photography is severely limited by itsphysical and impractical requirements of needing extremely large amountsof computer memory (both archival, disk-type memory and active,solid-state memory), needing extensive computational processing power,and requiring much time and user intervention to setup equipment andemit and capture multitudes of light rays for every projectionenvironment in which the projector-camera system is to be used.

A clearer understanding of dual photography may be obtained withreference to FIGS. 1A and 1B. In FIG. 1A, a “primal configuration” (i.e.a configuration of real, physical devices prior to any dualitytransformations) includes a real digital projector 11, a real projectedimage 13, and a real digital camera 15. Light is emitted from realprojector 11 and captured by real camera 15. A coefficient relating eachprojected light ray (from each projector pixel e within real projector11) to a correspondingly captured light ray (captured at each camerasensor pixel g within real camera 15) is called a light transportcoefficient. Using the light transport coefficient, it is possible todetermine the characteristics of the projected light ray from thecaptured light ray.

In the present example, real projector 11 is preferably a digitalprojector having a projector pixel array 17 symbolically shown as adotted box and comprised of s rows and r columns of individual projectorpixels e. Each projector pixel e may be the source of a separatelyemitted light ray. The size of projector pixel array 17 depends on theresolution of real projector 11. For example, a VGA resolution mayconsist of an array of 640 by 480 pixels (i.e. 307,200 projector pixelse), an SVGA resolution may have an array of 800 by 600 pixels (i.e.480,000 projector pixels e), an XVG resolution may have an array of 1024by 768 pixels (i.e. 786,732 projector pixels e), an SXVG resolution mayhave an array of 1280 by 1024 pixels (i.e. 1,310,720 projector pixelse), and so on, with greater resolution projectors requiring a greaternumber of individual projector pixels e.

Similarly, real camera 15 is a digital camera having a camera sensorpixel array 19 symbolically shown as a dotted box and comprised of urows and u columns of individual camera pixels g. Each camera pixel gmay receive, i.e. capture, part of an emitted light ray. The size ofcamera sensor pixel array 19 again depends on the resolution of realcamera 15. However, it is common for real camera 15 to have a resolutionof 4 MegaPixels (i.e. 4,194,304 camera pixels g), or greater.

Since each camera pixel g within camera sensor pixel array 19 maycapture part of an individually emitted light ray from a distinctprojector pixel e, and since each discrete projector pixel e may emit aseparate light ray, a multitude of light transport coefficients areneeded to relate each discrete projector pixel e to each, and every,camera pixel g. In other words, a light ray emitted from a singleprojector pixel e may cover the entirety of camera sensor pixel array19, and each camera pixel g will therefore capture a different amount ofthe emitted light ray. Subsequently, each discrete camera pixel g willhave a different light transport coefficient indicating how much of theindividually emitted light ray it received. If camera sensor pixel array19 has 4,194,304 individual camera pixels g (i.e. has a 4 MegaPixelresolution), then each individual projector pixel e will require aseparate set of 4,194,304 individual light transport coefficients torelate it to camera sensor pixel array 19. Therefore, millions ofseparately determined sets of light transport coefficients (one set perprojector pixel e) will be needed to relate the entirety of projectorpixel array 17 to camera sensor pixel array 19 and establish a dualityrelationship between real projector 11 and real camera 15.

Since in the present example, each discrete projector pixel e requires aseparate set of 4,194,304 individually determined light transportcoefficients to relate it to real camera 15, and since real projector 11may have millions of discrete projector pixels e, it is beneficial toview each set of light transport coefficients as a separate array oflight transport coefficients and to collect these separate arrays into asingle light transport matrix (T). Each array of light transportcoefficients constitutes a separate column within light transport matrixT. Thus, each column in T constitutes a set of light transportcoefficients corresponding to a separate projector pixel e.

Since in the present example, real projector 11 is a digital projectorhaving an array of individual light projector pixels e and real camera15 is a digital camera having an array of individual camera pixels g, alight transport matrix T will be used to define the duality relationshipbetween real projector 11 and real camera 15. In the followingdiscussion, matrix element T_(ge) identifies an individual lighttransport coefficient (within light transport matrix T) relating anindividual, real projector pixel e to an individual, real camera pixelg.

A real image, as captured by real camera 15, is comprised of all thelight rays individually captured by each camera pixel g within camerasensor pixel array 19. It is therefore helpful to organize a realcaptured image, as determined by camera sensor pixel array 19, into areal-image capture matrix, C. Similarly, it is beneficial to organize areal projected image, as constructed by activation of the individualprojector pixels e within projector pixel array 17, into a real-imageprojection matrix, P. Using this notation, a real captured image (asdefined by real-image capture matrix C) may be related to a realprojected image (as defined by real-image projection matrix P) by thelight transport matrix T according to the relationship, C=TP.

The duality transformation, i.e. dual configuration, of the system ofFIG. 1A is shown in FIG. 1B. In this dual configuration, real projector11 of FIG. 1A is transformed into a virtual camera 11″, and real camera15 of FIG. 1A is transformed into a virtual projector 15″. It is to beunderstood that virtual camera 11″ and virtual projector 15″ representthe dual counterparts of real projector 11 and real camera 15,respectively, and are not real devices themselves. That is, virtualcamera 11″ is a mathematical representation of how a hypothetical camera(i.e. virtual camera 11″) would behave to capture a hypotheticallyprojected dual image 13″, which is similar to real image 13 projected byreal projector 11 of FIG. 1A. Similarly, virtual projector 15″ is amathematical representation of how a hypothetical projector (i.e.virtual projector 15″) would behave to project hypothetical dual image13″ that substantially matches real image 13, as captured by real camera15 (of FIG. 1A). Thus, the positions of the real projector 11 and realcamera 15 of FIG. 1A are interchanged in FIG. 1B as virtual camera 11″and virtual projector 15″.

It should be noted that the pixel resolution of the real devices carriesforward to their counterpart virtual devices (i.e. dual devices).Therefore, virtual camera 11″ has a virtual camera sensor pixel array17″ consisting of s rows and r columns to match the resolution ofprojector pixel array 17 of real projector 11. Similarly, virtualprojector 15″ has a virtual projector pixel array 19″ consisting of urows and u columns to match the resolution of camera sensor pixel array19 of real camera 15.

If one assumes that dual light transport matrix T″ is the lighttransport matrix in this dual configuration such that a dual-imagecapture matrix C″ (which defines dual image 13″ as captured by virtualcamera 11″) relates to a dual-image projection matrix P″ (which definesdual image 13″ as projected by virtual projector 15″) as C″=T″P″, thenT″_(eg) would be an individual dual light transport coefficient relatingan individual virtual projector pixel g″ to an individual virtual camerapixel e″.

Helmholtz reciprocity specifies that the pixel-to-pixel transportcoefficient is equal in both directions (i.e. from real projector 11 toreal camera 15, and from virtual projector 15″ to virtual camera 11″).That is, T″_(eg)=T_(ge), which means T″=T^(T), (i.e. dual lighttransport matrix T″ is equivalent to the result of the mathematical,matrix transpose operation on real light transport matrix T). Therefore,given light transport matrix T, one can use T^(T) to synthesize thedual, or virtual, images that would be acquired in the dualconfiguration.

Thus, the light transport matrix T permits one to create images thatappear to be captured by a projector, with a camera acting as a secondprojector. However, as is explained above, the high complexity involvedin generating and manipulating light transport matrix T has heretoforegreatly limited its application, particularly in the field ofcalibrating projector-camera systems.

Other problems associated with projector-camera systems are how tocompensate for light diffusing objects that may obstruct a projector'sline of sight. Of related interest are issues of whetherprojector-camera systems can be used to achieve more complex images thantypical. For example, can such systems combine multiple images frommultiple projectors to create a single composite image? Alternatively,can one generate “3-D” images, or other visual effects that previouslyrequired more complex equipment and more complex projection setups?Also, can one make better use of the camera in a projector-camera systemso that the camera can be an active part of an image creation process.Furthermore, what are the implications of using a low resolution,inexpensive camera in such projector-camera systems?

Previous works [Raskar et al. 1998; Underkoffler and Ishii 1998] putforth the concept of intelligent illumination and showed how projectorscould be used to enhance workplace interaction and serve as novel toolsfor problem solving. The projector-camera community has since solvedmany of the technical challenges in intelligent projectors. Inparticular, significant advances have been made in automatic mosaicingof multiple projectors [Chen et al. 2002; Yang et al. 2001; Raij et al.2003; Sukthankar et al. 2001; Raskar et al. 2003].

[Raskar et al. 2001] demonstrated projection onto complex objects. Usingpreviously created 3D models of the objects; multiple projectors couldadd virtual texture and animation to real physical objects withnon-trivial complicated shapes.

[Fujii et al. 2005] proposed a method that modified the appearance ofobjects in real time using a co-axial projector-camera system.[Grossberg et al. 2004] incorporated a piecewise polynomial 3D model toallow a non-co-axial projector-camera system to perform view projection.

Projector camera systems have also been used to extract depth maps[Zhang and Nayar 2006], and space-time-multiplexed illumination has beenproposed as a means for recovering depth edges [Raskar et al. 2004].

As will be explained more fully below, the present invention addressesthe problem of how to determine what a projector needs to project inorder to create a desired image by using the inverse of the lighttransport matrix, and its application will further simplify thecalibration of projector-camera systems.

SUMMARY OF THE INVENTION

The above objects are met in a system that simplifies the generation oftransport matrix T, simplifies the implementation of light transportcoefficients in dual photography, modifies while still furthersimplifying the light transport matrix T to incorporate compensation forlight scattering effects or light noise. The above objects are furthermet in a system that uses the transport matrix T to simplify thecoordinating to two digital projectors at differing angles to projectindividual, but fully coordinated images designed to create a perceived3D image.

Application of dual photography is simplified by reducing the number ofcaptured images needed to generate a light transport matrix T of (p×q)projector pixel array from (p×q) images to (p+q) images. Manipulation ofthe light transport matrix T is simplified by replacing the use of afully populated light transport matrix T with an index associating eachprojector pixel to only non-zero light transport coefficient values. Byeliminating the use of zero-valued light transport coefficients, thememory and processing requirements for implementing dual photography aregreatly reduced. This dual photography technique is applied to thecalibration of projector-camera systems.

The present invention further applies various techniques to creation of3D images. More specifically, herein is shown a method of creating aperspective image (3-Dimensional image or 3D image) as perceived from apredefined reference view point, comprising: providing a first digitalprojector having a first projection region arranged along a firstorientation with reference to a reference base; providing a seconddigital projector having a second projection region arranged along asecond orientation with reference to said reference base, the secondprojection region at least partially overlapping the first projectionregion; defining a 3D imaging region within the overlapping area of thefirst projection region and the second projection region; establishing afirst light transport matrix T₁ covering the portion of the firstprojection region that encompasses the 3D imaging region, the firstlight transport matrix T₁ relating a first image c₁ as viewed from saidpredefined reference view point to a first projected image p₁ from thefirst digital projector according to relation c₁=T₁p₁; establishing asecond light transport matrix T₂ covering the portion of the secondprojection region that encompasses said 3D imaging region, the secondlight transport matrix T₂ relating a second image c₁ as viewed from thepredefined reference view point to a second projected image p₂ from saidsecond digital projector according to relation c₂=T₂p₂; providing afirst imaging sequence to said first digital projector, the firstimaging sequence providing a first angled view of an image subjectwithin said 3D imaging region with reference to said reference viewpoint as defined by said first light transport matrix T₁; providing asecond imaging sequence to the second digital projector, the secondimaging sequence providing a second angled view of the image subjectwithin the 3D imaging region with reference to the reference view pointas defined by the second light transport matrix T₂.

Further to this method, the first light transport matrix is T₁ isconstructed only for projector pixels of the first digital projectorthat contribute to the 3D imaging region.

This method also provides a pair of eyeglasses at the predefinedreference view point, the pair of eyeglasses having: a first polarizedlens for capturing light from the first digital projector and rejectinglight from the second digital projector; and a second polarized lens forcapturing light from the second digital projector and rejecting lightfrom the first second digital projector. The first orientation ispreferably at 90 degrees to the second orientation.

In the above method, the first digital projector preferably has a firstset of Red, Green, and Blue projection panels, two projection panelswithin the first set being polarized along a first polarizationorientation with reference to said reference base, and a thirdprojection panel within the first set being polarized along a secondpolarization orientation with reference to said reference base.Similarly, the second digital projector preferably has a second set ofRed, Green, and Blue projection panels, two projection panels within thesecond set being polarized along said second polarization orientationwith reference to the reference base, and a third projection panelwithin the second set being polarized along the first polarizationorientation with reference to the reference base. In this case, themethod includes identifying first imaging information within the firstimaging sequence corresponding to the third projection panel within thefirst set, and transferring to the second digital projector the firstimaging information for projection by the second digital projector. Themethod can then identify second imaging information within the secondimaging sequence corresponding to the third projection panel within thesecond set, and transfer to the first digital projector the secondimaging information for projection by the first digital projector.

Another approach to the method is when the first imaging sequence has R1(Red 1), G1 (Green 1), and B1 (Blue 1) imaging channels, and the secondimaging sequence has R2 (Red 2), G2 (Green 2), and B2 (Blue 2) imagingchannels. In this case, one can swap the R1 and R2 channels of saidfirst and second imaging sequences.

The above described method, the establishing of the first lighttransport matrix T₁ can include using a camera at the predefinedreference view point to capture a reference noise image C₀ of at leastpart of the first projection region under ambient light with no imageprojected from the first digital projector. One then uses the camera atthe predefined reference view point to capture a first image of a firsttest pattern projected by the first digital projector within the firstprojection region. The first test pattern is preferably created bysimultaneously activating a first group of projection pixels within thefirst digital projector, with all projection pixels not in the firsttest pattern being turned off. Again, using the camera at saidpredefined reference view point, one may capture a second image of asecond test pattern projected by the first digital projector within thefirst projection region. In this case, the second test pattern iscreated by simultaneously activating a second group of projection pixelswithin the first digital projector, with all projection pixels not inthe second test pattern being turned off. It is noted that the first andsecond groups of projection pixels preferably have only one projectionpixel in common defining a target projection pixel. One then maysubtract the reference noise image C₀ from the a first image to make afirst corrected image, subtract the reference noise image C₀ from thesecond image to make a second corrected image. Afterwards, one cancompare image pixels of the first corrected image to corresponding imagepixels of the second corrected image and retaining the darker of twocompared image pixels, the retained image pixels constituting acomposite image. One then identifies all none-dark image pixels in thecomposite image, and selectively identifies the none-dark pixels in thecomposite image as non-zero light transport coefficients associated withthe target projection pixel.

Preferably, only pixels within the composite image having a lightintensity value not less than a predefined minimum value are identifiedas the none-dark image pixels. In this case, the predefined minimumvalue is preferably 25% of the maximum light intensity value.

Also in this method, identified non-dark image pixels are preferablyarranged into adjoining pixel cluster according to their pixel positionrelation within an image, and only pixels within a cluster having a sizenot less than a predefined number of pixels are identified as non-darkpixels. The predefined number of pixels can be defined by the number ofpixels within an average sized light footprint created on thecamera-pixel-array within the camera created by turning ON a singleprojection pixel within the first digital projector. In this case, thecentroid within a cluster having a size not less than the predefinednumber of pixels constitutes an estimate of the camera pixel arraycoordinate corresponding to the target projection pixel.

The present method may also include: (a) defining a desired compositeimage “c”; (b) setting p₂ equal to zero, solving for p₁ in formula p₁=T₁⁻¹ (c−T₂p₂); (c) using the current solved value for p₁, solving for p₂in formula p₂=T₂ ⁻¹(c−T₁p₁); (d) using the current solved value for p₂,solving for p₁ in formula p₁ T₁ ⁻¹(c−T₂p₂); and repeating steps (c) and(d) in sequence until p₁ converges to a first mosaicing image and p₂converges to a second mosaicing image. In this case, the desiredcomposite image “c” can preferably be defined as “c” =c₁+c₂. Also, atlease at least one of T₁ ⁻¹ or T₂ ⁻¹ is an estimate of an inverse of ageneral light transport matrix T, and the estimate is generated by:identifying in turn, each column in the general light transport matrix Tas a target column, calculating normalized values for not-nullifiedentry values in the target column with reference to the target column;creating an intermediate matrix of equal size as the general lighttransport matrix T; populating each column in the intermediate matrixwith the calculated normalized values of its corresponding target columnin the general light transport matrix T, each normalized value in eachpopulated column in the intermediate matrix maintaining a one-to-onecorrespondence with the not-nullified entry values in its correspondingcolumn in the general light transport matrix T; and applying a transposematrix operation on the intermediate matrix.

Following this approach, if the intermediate matrix is denoted as {hacekover (T)}, a target column in general light transport matrix T isdenoted as Tr and a corresponding column in {hacek over (T)} is denotedas {hacek over (T)}r, then the construction and population of {hacekover (T)} is defined as {hacek over (T)}r=Tr/(∥Tr∥)² such thatp₁=({hacek over (T)}₁)^(T)(c−T₂p₂) if T₁ ⁻¹ is the estimate of theinverse of general light transport matrix T⁻¹ and p₂=({hacek over(T)}₂)^(T)(c−T₁p₁) if T₂ ⁻¹ is the estimate of the inverse of a generallight transport matrix T⁻¹. Preferably, at least one of T₁ or T₂ is anestimated light transport matrix generated from a sampling of lightfootprint information not having a one-on-one correspondence with everyprojector pixel in its respective digital projector.

Other objects and attainments together with a fuller understanding ofthe invention will become apparent and appreciated by referring to thefollowing description and claims taken in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings wherein like reference symbols refer to like parts.

FIGS. 1A and 1B show a prior art setup for implementation of dualphotography.

FIGS. 2A and 2B show a setup for dual photography in accord with thepresent invention.

FIG. 3A is an example of a light ray projected from a single projectorpixel, bouncing off a scene, and forming a small light footprint on acamera sensor pixel array.

FIG. 3B is an example of how a projected image may be represented as anarray of information.

FIG. 3C is an example of how a captured light footprint on a camerasensor pixel array may be represented as an array of information.

FIG. 4A is an illustrative example of a projected footprint on a lightsensor array within a digital camera resulting from activation of asingle projector pixel in a projector.

FIG. 4B is an illustrative example of a column of light transfercoefficients within a matrix T reflecting the example of FIG. 3A.

FIGS. 5A and 5B show two examples of two columns of projector pixelssimultaneously projected onto a scene having a checkerboard pattern.

FIGS. 6A and 6B show two examples of two rows of projector pixelssimultaneously projected onto a scene having a checkerboard pattern.

FIG. 7A shows a generated light ray footprint resulting from a singleprojector pixel, as created by combining the images of FIGS. 5A and 6A.

FIG. 7B shows a generated light ray footprint resultant from a singleprojector pixel, as created by combining the images of FIGS. 5B and 6B.

FIG. 8 is a first example of an index associating projector pixels tonon-zero valued light transport coefficients, as determined by a lightray footprint as generated in FIG. 7A or 7B.

FIG. 9 is second example of an index associating projector pixels tonon-zero valued light transport coefficients, as determined by a lightray footprint as generated in FIG. 7A or 7B.

FIG. 10A shows a real captured image taken by a real camera.

FIG. 10B shows a dual captured image, as seen by a real projector, asgenerated using the method of the present invention.

FIG. 11 is an example of a result obtained by the use of homography tocalibrate a projector.

FIG. 12 shows a projected image distorted by two wine glasses placedbetween a projector and a scene.

FIG. 13 is an example of two overlapping adjacent light footprintsproduced by two distinct projector pixels.

FIG. 14 is an example of a process in accord with the present inventionfor imposing the display constraint on an arbitrary scene.

FIG. 15 is a further step in the imposing of the display constraint asintroduced in FIG. 14.

FIG. 16 is an example of the present invention being applied to colorcompensation in poster images.

FIG. 17 shows the projection scene of FIG. 12, but with the presentinvention's method of compensating for light distortion applied.

FIG. 18 is an exemplary projection setup for using dual photography tocreate an immersive display system.

FIG. 19 shows an immersive projector P2 being used to simulate a frontprojector P1.

FIG. 20 is an example of an image generated using the virtual projectorimplementation of FIG. 19.

FIG. 21A shows the right side of an image projected by a real frontprojector.

FIG. 21B shows the corresponding left side of the image shown in FIG.21A, but in FIG. 21B the left side of the image is projected by animmersive projector.

FIG. 21C shows the right side image of FIG. 21A joined to the left sideimage of FIG. 21B.

FIGS. 22A and 22B show two additional examples of a left side imagegenerated by an immersion projector joined to the right side imagegenerated by a front projector.

FIGS. 23A to 23C show an alternate application of the present inventionto recreate in a real room a virtual image created in a virtual modelroom.

FIG. 24 is an example of the application of the technique of FIGS.23A-23C to project an image bigger than the projection space in a realroom without image distortion.

FIG. 25 is an exemplary projection system in accord with the presentinvention in a minimal form.

FIG. 26 shows a prototype based on the design of FIG. 25.

FIG. 27 is an alternate view of the setup of FIG. 26.

FIG. 28A shows, under ambient lighting, a room with the projectionsystem of FIGS. 26 and 27 installed.

FIG. 28B shows the room of FIG. 28A under immersive projection lighting.

FIG. 29A shows the room of FIG. 28A with an uncalibrated projectedimage.

FIG. 29B shows the room of FIG. 28A with a calibrated projected image.

FIG. 30A shows the room of FIG. 28A with an uncalibrated blankprojection.

FIG. 30B shows the room of FIG. 28A with a calibrated blank projection.

FIG. 31 is a first step in the application of the present invention to adome mirror projector.

FIG. 32 is a second step in the application of the present invention toa dome mirror projector.

FIG. 33 is a third step in the application of the present invention to adome mirror projector.

FIG. 34 demonstrates the application of the present invention to adesired image, to produce a precisely distorted image for projection ona dome mirror projector.

FIG. 35 demonstrates the result of projecting the distorted image ofFIG. 34.

FIG. 36 is an alternative design for ceiling-mounted operation.

FIG. 37 is an alternate configuration of the present invention.

FIG. 38 is still another configuration of the present invention.

FIG. 39 shows a set up for mosaicing two, or more, projector-camerasystems to create composite image.

FIG. 40 shows a composite of images from multiple projector-camerasystems.

FIG. 41 shows an image as projected by a first of two projector-camerasystems in a multi-projector-camera system.

FIG. 42 shows an image as projected by a second of two projector-camerasystems in a multi-projector-camera system.

FIG. 43 is an example of the present invention applied to the projectionof a checkerboard pattern onto a complex surface.

FIG. 44A shows a design that uses a single curved mirror 125 andmultiple projector-camera pairs 145.

FIG. 44B shows a design that uses a single mirror pyramid 151 andmultiple projector-camera pairs 145.

FIG. 45 shows how multiple large FOV projectors 153 a and 153 b can beused to achieve an even larger overall projection FOV.

FIG. 46 shows an example configuration of two projectors oriented so asto project images with different polarization to achieve a 3D visualimage.

FIG. 47 is a symbolic representation of the configuration of FIG. 46.

FIG. 48 is a symbolic representation of the pair of polarized glassesshown in the configuration of FIG. 46.

FIG. 49 illustrates the swapping of the R component of respective RGBimages between two projectors.

FIG. 50 illustrates a desired uniform polarization of two projectedimages from two respective projectors.

FIG. 51 illustrates creation of 3D visual content within an intersectionregion of two projected images.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

In order to apply camera calibration techniques to projectors, one wouldrequire projectors to be able to capture images. That is, if projectorscould capture images, then projector-camera systems could be treatedlike multi-camera systems, and standard camera calibration techniques(described above) might be used to calibrate projector-camera systems.In other words, if a projector could be treated as a pseudo-camera, thenit could be calibrated along with a real camera in a manner similar tothe camera calibration stage of the multi-camera system described above,and the “bootstrapping” projector calibration stage previously used forcalibrating projector-camera systems might be eliminated.

With reference to FIG. 2A, an imaging setup in accord with the presentinvention may include a real projector 21 and a real camera 25. Realprojector 21 is preferably a digital projector and has an imagingelement including an imaging projection array (i.e. projector pixelarray 27), consisting of p rows and q columns of individual imagingprojection elements (i.e. projector pixels j) Projector pixel array 27is internal to real projector 21, and is shown for discussion purposesas crossed lines within a dotted square in FIG. 2A. Real projector 21 ispreferably of the liquid crystal display (LCD) type, digital lightprocessing (DLP) type, liquid crystal on silicon (LCOS) type, or otherdigital projection technology type.

Preferably, real camera 25 is a digital camera having an image sensorincluding a camera sensor pixel array 29 (i.e. light receptor array orimage sensor array), consisting of m rows by n columns of individualcamera pixels i (i.e. image sensor elements or light receptor pixels).For simplicity, camera sensor pixel array 29 is shown on real camera 25,but it is to be understood that camera sensor pixel array 29 is internalto real camera 25.

This physical setup using real projector 21 and real camera 25 ispreferably called the ‘primal’ setup. Light rays emitted from individualprojector pixels j within real projector 21 form real image 23 bybouncing off a projection surface (i.e. display environment or scene),which may have an irregular or flat shape, and some of the light rayseventually reach the image sensor within real camera 25. In general,each light ray is dispersed, reflected, and refracted in the scene andhits the camera's image sensor at a number of different locationsthroughout camera sensor pixel array 29. Thus, when a single light rayemitted from an individual imaging projector pixel j from real projector21 reaches real camera 25, the individually projected light ray forms anm-by-n image on camera sensor pixel array 29, with each individualcamera pixel i receiving a certain amount of light intensitycontribution from the projected light ray.

Consequently, although it might appear that ideally a single lighttransport coefficient relating one projector pixel j to one camera pixeli might be determinable by individually turning ON a single projectorpixel j to emit a single light ray to hit a single camera pixel i, thisis not the case. In reality, the entire camera sensor pixel array 29will receive a different contribution of light intensity from the singlyemitted light ray. Therefore, each light ray emitted from eachindividual projector pixel j generates a different set, or array, ofindividual light transport coefficients, one for each camera pixel iwithin camera sensor pixel array 29. Consequently, each set (i.e. array)will consist of (m×n) [i.e. m-multiplied-by-n] individual lighttransport coefficients, one for each camera pixel i.

If each set of light transport coefficients is arranged as a column ofcoefficients to form a composite light transport matrix, T, then thecomposite light transport matrix T will have a different column of lighttransport coefficients for each individual projector pixel j.Furthermore, since there is a one-to-one correspondence between eachlight transport coefficient entry (i.e. matrix element) within eachcolumn and each camera pixel i, each column represents the entire imagecaptured at camera 25 resulting from a single projector pixel j beingturned ON. Accordingly, the entire (i.e. composite) light transportmatrix T will consist of (p×q) [i.e. p multiplied-by q] columns (onecolumn [i.e. captured image] for each individually turned ON projectorpixel j) and (m×n) rows (one row for each individual camera pixel i).

In the following discussion, it is beneficial to view a real projectedimage as a first vector, Rprjct, (i.e. real projection vector) having(p×q) elements (one for each projector pixel j), and a resultant realcaptured image as a second vector, Rcptr, (i.e. real captured imagevector) having (m×n) elements (one for each camera pixel i).

Using the notation that a real projected image (i.e. an image projectedusing the entire projector pixel array 27) is represented as a (p×q)real projection vector “Rprjct” (i.e. a “p-by-q vector”), and thenotation that a correspondingly real captured image (i.e. an imagecaptured by the entire camera sensor pixel array 29) is represented asan (m×n) real captured image vector “Rcptr” (i.e. an “m-by-n vector”),then the light transport relationship between real projector 21 and realcamera 25 can be written as

Rcptr=T Rprjct

where T is a composite light transport matrix that relates realprojector 21 to real camera 25. It is to be understood that lighttransport matrix T would have been previously generated by, for example,individually and separately turning ON each projector pixel j anddetermining the corresponding light transport coefficient for eachindividual camera pixel i.

To recapitulate, since each projector pixel j results in a light raythat is scattered across the entire camera sensor pixel array 29, eachindividual camera pixel i will have a differently valued light transportcoefficient indicative of an intensity value of a predefined lightcharacteristic it received from each individual projector pixel j. Inthe present embodiments, this light characteristic is preferably ameasure of light intensity. Therefore, each projector pixel j willresult in a column of (m×n) individual light transport coefficients,each coefficient indicating an amount of light intensity received byeach camera pixel i. Since real projector 21 has (p×q) projector pixelsj, light transport matrix T will have (p×q) columns [one for eachprojector pixel j] and (m×n) rows [one for each camera pixel i] ofindividual light transport coefficients. Thus, light transport matrix Thas traditionally been necessarily huge, consisting of (p×q×m×n)individual light transport coefficients.

With reference to FIG. 2B, a “dual” setup is one where real projector 21is replaced by a virtual camera 21″ having a virtual camera sensor pixelarray 27″ of equal size as real projector pixel array 27. Thus, virtualcamera 21″ has a virtual camera sensor pixel array 27″ comprised ofp-rows by q-columns of virtual camera pixels j″. Similarly, in this“dual” setup, real camera 25 is replaced by a virtual projector 25″having a virtual projector pixel array 29″ of equal size as real camerasensor pixel array 29. Therefore, virtual projector 25″ has a virtualprojector pixel array 29″ comprised of m-rows by n-columns of virtualprojector pixels i″.

In this case, a virtual image 23″ (as projected by virtual projector25″) would be represented by an (m×n) virtual projection vector,Vprjct″. Similarly, a virtually captured image captured by virtualcamera 21″ would be represented by a (p×q) virtual captured imagevector, Vcptr″. Therefore in this dual setup, virtual captured imagevector Vcptr″ relates to virtual projection vector Vprjct″ by a “dual”light transport matrix T″. By the principle of Helmholtz reciprocity,the light transport is equal in both directions. Therefore, the duallight transport matrix T″ for the dual setup (i.e. the dualitytransformation setup) relates virtual captured image vector Vcptr″ tovirtual projection vector Vprjct″ by following relation:

Vcptr″=T″Vprjct″

This relationship, however, brings up the problem of how can onedetermine the multitudes of “virtual” light transport coefficients thatmake up dual light transport matrix T″, and that relate a virtuallyprojected image to a virtually captured image? Remarkably, it has beenfound that a duality transformation from a real light transport matrix,T, to its dual light transport matrix, T″, can be achieved by submittingthe real light transport matrix T to a transpose matrix operation. Thus,T″ is the transpose of T, such that T″=T^(T) and Vcptr″=T^(T) Vprjct″.

As it is known in the art of matrix computation, the transpose matrixoperation of a general [x×y] matrix A is denoted by the addition of asuperscript letter “T” (such as A^(T), for example) and is defined by an[y×x] matrix whose first column is the first row of matrix A, and whosesecond column is the second row of matrix A, whose third column is thethird row of matrix A, and so on. As is readily evident, this matrixoperation simply flips the original matrix A about its first element,such that its first element (i.e. at position (1,1)) remains unchangedand the bottom of the first column becomes the end of the first row.Consequently, if one can capture, or otherwise determine, the real lighttransport matrix T for a primal setup, then the dual light transportmatrix T″ for the dual setup is readily computable by flipping the reallight transport matrix T to obtain its transpose, T^(T), as described.In further discussions below, dual light transport matrix T″ andtransposed light transport matrix T^(T) may be used interchangeably.

It is noted, however, that if one can provide a simplified method ofdetermining T, one can directly determine T^(T) from T for the dualitytransformation. Before proposing a method for simplifying thedetermination of T, it is beneficial to first discuss some of thedifficulties of traditional methods of determining T.

As is explained above, real light transport matrix T holds theindividual light transport coefficients corresponding between eachindividual projector pixel j of real projector 21 and all the individualcamera pixels i of real camera 25. Therefore, a determination of eachindividual light transport coefficient corresponding between anindividual projector pixel j and all camera pixels i should avoid lightray contributions from other projector pixels j in projector pixel array27.

To accomplish this, one may first consider an initial, real projectionimage_j created by setting a j^(th) projector pixel to a value of 1(i.e. is turned ON) and setting to a zero value (i.e. turning OFF) allother elements in projector pixel array 27. In this case, the j^(th)projector pixel is the projector pixel under-test for which the lighttransport coefficients is to be determined. These test conditions areannotated as a real projection vector Rprjct_j that has a value of oneat an entry location associated with the j^(th) projector pixel, and hasall other entry locations corresponding to all other projector pixelsset to a value of zero. One may then capture real projection image_j todetermine its corresponding (m×n) real captured image vector Rcptr_j,which defines the j^(th) column of matrix T.

This method of acquiring a column of light transport coefficients formatrix T for a given j^(th) projector pixel, suggests that a systematicmethod for capturing the entire matrix T is to sequentially turn ON eachprojector pixel j of real projector 21 (one projector pixel at a time),and to capture its corresponding real image, Rcptr_j, with real camera25. When all p×q projector pixels j have been sequentially turned ON,and their corresponding real images Rcptr_j have been captured, all thecaptured image vectors Rcptr_1 to Rcptr_(p×q) are grouped to form matrixT. Each captured image vector Rcptr_j constitutes a column of lighttransport coefficient entries in matrix T. This results in a matrix Thaving (p×q) columns and (m×n) rows of individual light transportcoefficients.

This straight forward, and systematic process for determining matrix T,however, is obviously a time-consuming process requiring (p×q) imageprojection-and-capture steps. Furthermore, the resultant light transportmatrix T is very large, consisting of (p×q×m×n) elements. Because of thesize of matrix T, computing a dual image is an extremely computationintensive operation requiring matrix multiplication between dual lighttransport matrix, T^(T), (which has the same number of elements asmatrix T), and a virtual projection vector Vprjct″ (which is a longvector having (m×n) elements).

In the past, a scheme for determining matrix T by adaptively turning onmultiple pixels of a real projector at a time (such as projector 11 ofFIG. 1A), has been suggested to speed up the process of determiningmatrix T. However, this scheme is complex, and requires that an image tobe projected be divided into multiple zones, and that one projectorpixel from each zone be selected. All selected projector pixels from themultiple zones are turned on simultaneously while assuring that thesimultaneously lit projector pixels are selected distant enough fromeach other to eliminate light interference between each other.Consequently, this scheme is scene-dependent (i.e. dependent upon thedisplay environment) since it is necessary to assure that lightinterference between individually selected projector pixels isminimized. Since creation of the multiple zones is dependent on thedisplay environment, this scheme requires physical inspection of aprojection area and manual setup, and therefore does not easily lenditself to general use or to automation.

A feature of the present invention proposes a method of reducing thenumber of procedural steps in the determination of real light transportmatrix T. That is, instead of requiring (p×q) imageprojection-and-capture steps (and the storing of p×q captured images),the presently proposed method captures at most “p plus q” [i.e. (p+q)]images, and under specific circumstances, described below, the numbercan be further reduced dramatically.

The present method is based on the following assumption: for mostprojector-camera display applications, any two distinct light rays b andc emitted from real projector 21 will typically hit camera sensor pixelarray 29 at distinct parts. That is, any light ray emitted from aprojector pixels j will not be dispersed across the entirety of camerasensor pixel array 29, but will be primarily concentrated within a smallgroup of camera pixels i forming a light footprint within a distinctregion of camera sensor pixel array 29. Furthermore, there will be no,or negligible, overlap in adjacent light footprints (i.e. negligibleoverlap in the camera pixels i hit by distinct projected light rays fromdistinct projector pixels j). This ideal light ray distributioncondition may be assumed when the resolution of the real camera 25 ismuch higher (i.e. at least two times higher) than the resolution realprojector 21.

This assumption regarding the non-overlap of distinct light rays, offcourse, is not true in general. For example, the ideal light raydistribution condition may not be achievable when the resolution of realcamera is 25 is comparable to, or smaller than, the resolution of realprojector 21, or when a light diffusing material is located between thelight path of a projected light ray from real projector 21 to camerasensor pixel array 29. For example, if the projection scene (i.e.display environment or projection surface/area) includes light diffusingmaterial, such as a glass of milk, the projected light rays will bediffused and the likelihood of significant light overlap between thedifferent light rays at camera sensor pixel array 29 is greatlyincreased. However, a display setup designed to ensure high resolutionprojections is likely to be devoid of such light diffusing material, andit is virtually guaranteed that each projected light footprint will besubstantially distinct from the next. That is, in venues, or settings,where high resolution projections are desired, it is likely that thevenue will be clear of light diffusing articles along the light path ofa projected image.

In the present first embodiment, it is assumed that the pixel resolutionof real camera 25 is at least four times greater than that of realprojector 21 and that the projection scene is devoid of any lightdiffusing material, such that the requirements for the ideal light raydistribution condition are met. Nonetheless, alternative embodiments formore general situations where the resolution of real camera 25 is notrestricted to be greater than real projector 21, or where the light pathbetween real projector 21 and real camera 25 is not limited to be devoidof light diffusing material are provided below.

A first pictorial representation of the present assumptions, and vectorgeneration method is shown in FIGS. 3A to 3C. With reference to FIG. 3A,an illustrative example of the present invention shows real projectorpixel array 27 having a pixel resolution smaller than that of realcamera sensor pixel array 29. For ease of discussion, projector pixelarray 27 is illustratively shown to consist of only 6 rows and 7 columns(i.e. p=6 and q=7) for a total resolution of 42 projector pixels j (i.e.(6×7) elements). Similarly for ease of discussion, camera sensor pixelarray 29 is illustratively shown to consist of only 12 rows and 14columns (i.e. m=12 and n=14) for a total resolution of 168 camera pixelsi (i.e. (12×14) elements). It is noted that the resolution of camerasensor pixel array 29 is four times greater than the resolution ofprojector pixel array 27, and no light diffusing material is locatedwithin the light path from projector pixel array 27 to camera sensorpixel array 29, and so the ideal light ray distribution conditions forthe present first embodiment are met.

In FIG. 3A, each square within projector pixel array 27 represents adifferent projector pixel j. In the present example, it is desired toobtain the light transport coefficients for the projector pixel jlocated at the intersection of the third row and fifth column (i.e.projector pixel j=(5,3) is the projector pixel under test). As explainedabove, light diffusion, or light scattering, is ignored. Thus, projectorpixel j=(5,3) is shown to be turned ON, i.e. lit, as is indicated by awhite square, while all other projector pixels are shown to remainturned OFF, as indicated by dark gray squares. Projector pixel j=(5,3)emits light ray 1, which passes through a projector lens 3 beforehitting a projection surface 5 (which is basically a generic projectionscene determined by the environment in which real projector 21 and realcamera 25 are located). Since light diffusion is ignored in the presentexample, light ray 1 bounces off projection surface 5 and passes througha camera lens 7 before hitting a group of camera pixels i forming alight ray footprint indicated by circle Ft1. Since the resolution ofeach projector pixel j is much coarser than that of the camera pixels i,the created light ray footprint Ft1 on camera sensor pixel array 29covers several camera pixels i, with the camera pixels at the center oflight ray footprint Ft1 (indicated as white squares) receiving the mostlight intensity, and the camera pixels i along the periphery of lightray footprint Ft1 (indicated as light gray squares) receiving less lightintensity, and those camera pixels i not within light ray footprint Ft1(indicated as dark gray squares) receiving no light. The light transportcoefficients for each camera pixel i corresponding to the projectorpixel under test j=(5,3) may then be determined by determining the lightintensity reading of all camera pixels i within camera sensor pixelarray 29.

As is explained above, it is beneficial to view projector pixel array 27as a first vector (real projection vector Rprjct), and to view theresultant captured image across the entirety of camera sensor pixelarray 29 as a second vector (real captured image vector Rcptr). Apictorial representation of how real projector vector Rprjct may beconstructed is shown in FIG. 3B. Projector pixel array 27 is firstseparated into six rows, row_1 to row_6, and each row is then turned 90degrees to form six column segments col_seg_r1 to col_seg_r6. The sixcolumn segments col_seg_r1 to col_seg_r6 are then joined end-to-end toform one composite column col_1 consisting of (p×q) elements (i.e. (6×7)or 42 elements). In the present example, element col_1_row_19 (i.e. the19^(th) element down from the top of column col_1) corresponds to aprojector pixel j at array location (5,3) of projector pixel array 27.Element col_1_row_19 is shown as a white square, while all otherelements are shown as darkened squares, to indicate that elementcol_1_row_19 is the only projector pixel turned ON.

A similar process is applied to camera sensor pixel array 29 in FIG. 3Cto construct real captured image vector Rcptr. In the present exemplarycase, camera sensor pixel array 29 would be broken up into m rows (i.e.12 rows) that are each turned 90 degrees to form 12 column segments, notshown. In a manner similar to how projector pixel array 27 wasrearranged in FIG. 3B, the 12 column segments are joined end-to-end toform one composite column cmra_col_1 (i.e. vector Rcptr) of (m×n)elements (i.e. (12×14) or 168 elements).

In the present example, the light ray emitted from projector pixel j atarray location (5,3) of projector pixel array 27 is assumed to formlight ray footprint Ft1 on camera sensor pixel array 29. The span oflight ray footprint Ft1 is indicated by a circle, and encompasses twelvecamera pixels i on four different rows of camera sensor pixel array 29.The camera pixels i within light ray footprint Ft1 receive at least apart of the light intensity from a light ray emitted from projectorpixel j=(5,3) (i.e. matrix element col_1_row 19) from FIG. 3A or 3B. Thecamera pixels at the center of light footprint circle Ft1 receive themost light intensity, and are identified as white squares, and thecamera pixels along the perimeter of light footprint circle Ft1 receiveless light intensity and are identified as light gray squares. Thosecamera pixels not within light ray footprint Ft1 are shown as dark graysquares. As is illustrated in FIG. 3C, these white and light graysquares constitute nonzero elements within the captured image, and areshown as white and light gray squares (i.e. nonzero) valued NZ elementsinterspersed between many zero valued elements (i.e. dark gray squares)within vector Rcptr (i.e. column cmra_col_1).

A second example providing a close-up view of how the light footprint ofa single light ray from a projector pixel j may cover several camerapixels i is shown in FIGS. 4A and 4B. In FIG. 4A, a partial view ofanother exemplary camera sensor pixel array 29 shows individual camerapixels i numbered horizontally from 1 to n on the first row, continuingwith (n+1) to (2n) on the second row, and (2n+1) to (3n) on the thirdrow, and so on. Following this sequence, it is to be understood thatcamera pixels i along the bottom-most row would be numbered from(m−1)n+1 to (mn).

A second light ray footprint Ft2 of a single light ray from anotherexemplary, single, projector pixel j impacting camera sensor pixel array29 is denoted as a circle. For illustration purposes, those camerapixels i not within light ray footprint Ft2 [i.e. those camera pixels inot hit by the single light ray emitted from the j^(th) projector pixel]are shown as deeply darkened, those camera pixels i partly covered byfootprint Ft2 are shown as lightly darkened, and those camera pixels icompletely within footprint Ft2 are shown as having no darkening. As itis known in the art, each camera pixel i that is at least partiallycovered by light ray footprint Ft2 will register a light intensity valueproportional to the amount of light it receives. This light intensityvalue may be assigned as the light transfer coefficient for thatindividual camera pixel i. Alternatively, the light transportcoefficient of each camera pixel i may be made proportional to the lightintensity value registered by the individual camera pixel i.Nonetheless, those camera pixels i that are not directly hit by theprojection light ray from the j^(th) projector pixel should have a lightintensity value of zero (or close to zero, or below a predefinedthreshold light intensity value), and their corresponding lighttransport coefficient should likewise have a value of zero (or close tozero).

With reference to FIG. 4B, an example of a captured image vector Rcptr_j[or j^(th) column of matrix T], as would correspond to the footprint Ft2example of FIG. 4A is shown. This j^(th) column of matrix T isillustratively shown as a numbered vertical sequence of light transportcoefficients, each corresponding to the numbered camera pixel i ofcamera sensor pixel array 29 of FIG. 4A. The numerical sequence ofcapture vector Rcptr_j preferably follows the horizontally numberedsequence of individual camera pixels i in camera sensor pixel array 29shown in FIG. 4A. As shown, only those elements within captured imagedvector Rcptr_j that correspond to camera pixels i covered by light rayfootprint Ft2 have non-zero values, i.e. “NZ”, for light transportcoefficients. All other camera pixels have “ZERO” values for lighttransport coefficients. It is to be understood that “NZ” represents anynon-zero light transport coefficient value, and that this value would berelated to the amount of light intensity received by the correspondingcamera pixel i. Since light ray footprint Ft2 spans several rows ofcamera sensor pixel array 29, each row is sequentially listed incaptured image vector Rcptr_j, with several long sequences of zerovalued light transport coefficients interspersed between a few non-zero,NZ, valued light transport coefficients.

Returning now to the present novel method for determining matrix T, itis first noted that individual light transport coefficients forprojector pixel j map to column j of the light transport matrix T.Assuming minimal overlap between projector pixels, it follows that afirst set of projector pixels S1 within imaging projector pixel array 27maps to a corresponding set of columns (one per projector pixel) inlight transport matrix T.

-   -   [i.e. S1 ⊂ {1, . . . , (p×q)}]        Furthermore, it is assumed that the first set of projector        pixels S1 includes target projector pixel j, i.e. the target        projector pixel under test.

Let Rcptr_S1 be a first real image captured by real camera 25 of aprojected image created by the simultaneous activation of the first setof projector pixels S1.

Consider now a second set of projector pixels S2 who share onlyprojector pixel under test j in common with the first set of projectorpixels S1,

-   -   [i.e. S1 ∩ S2={j}]        Let Rcptr_S2 be a second real image captured by real camera 25        of a projected image created by the simultaneous activation of        the second set of projector pixels S2. The light transport        coefficients of the j^(th) column of light transport matrix T        (which corresponds to the target projector pixel under test,        i.e. corresponds to j) may be directly obtain from real captured        images Rcptr_S1 and Rcptr_S2 by identifying the one light ray        footprint they share in common (i.e. similar to light ray        footprints Ft1 or Ft2 in FIGS. 3A, 3C or 4A). This common light        ray footprint would correspond to a light ray emitted from        target projector pixel J, which is the only lit projector pixel        j shared in common among first set S1 and second set S2.

The next step, therefore, is to determine how to identify the one lightray footprint commonly shared by both real captured images Rcptr_S1 andRcptr_S2. A method of identifying this common light ray footprint is toconduct a pixel-by-pixel comparison of both captured images Rcptr_S1 andRcptr_S2, and to identify the dimmer of two compared pixels. Forexample, in first captured image Rcptr_S1 only sensor pixels withinindividual light ray footprints, each corresponding to the simultaneouslighting of the first set of projector pixels S1, will have non-zero(NZ) light intensity values, and all other pixels in captured imageRcptr_S1 will have zero values, i.e. will be comparatively dark.Similarly in second captured image Rcptr_S2, only sensor pixels withinlight ray footprints corresponding to the second set of simultaneouslyturned on projector pixels S2 have non-zero (NZ) light intensity values,and all other pixels will have zero (or dark) values (i.e. below apredefined threshold value). Since the two sets S1 and S2 share only thetarget projector pixel j in common, a direct comparison of both capturedimages will quickly identify the camera pixels i within the light rayfootprint corresponding to projector pixel j by identifying the onlynon-zero region (i.e. non-dark region) common to both Rcptr_S1 andRcptr_S2. Stated differently, the intersection of the lit regions (i.e.light ray footprints) of Rcptr_S1 and Rcptr_S2 is identified, and thisidentified light ray footprint will correspond to the target projectorpixel under-test, j.

A method of accomplishing this is to conduct a pixel-by-pixel comparisonof both captured images Rcptr_S1 and Rcptr_S2, and to retain only thedarker (i.e. dimmer) of the two compared pixels. This process may beexpressed as:

Tj≅MIN(Rcptr_S1, Rcptr_S2)

where Tj is the j^(th) column of matrix T, and “MIN” indicates that thelower valued camera pixel (i.e. the darker camera pixel having a lowercaptured light intensity value) in Rcptr_S1 and Rcptr_S2 is retained,and the higher valued (i.e. brighter) camera pixel is discarded. In thisway, the only high intensity values that are retained correspond to alight ray footprint common to both S1 and S2.

Stated differently, since the contribution of each individual projectorpixel j is mapped to distinct parts of the camera sensor pixel array 29,there is a set L of cameral pixels i among pixels i=1 to i=(m×n) commonto captured image Rcptr_S1 and Rcptr_S2 that corresponds to the targetprojector pixel, j.

-   -   [i.e. L ⊂ {1, . . . , (m×n)}]        It should again be noted that the target projector pixel, j, is        the intersection of projector pixel sets S1 and S2, (i.e. j is        the only projector pixel common to both sets S1 and S2), such        that    -   S1 ∩ S2={j}        Therefore, among the captured camera pixels (in both Rcptr_S1        and Rcptur_S2) that do not correspond to the target projector        pixel, j, (i.e. those camera pixels not in set L, i.e. ∉L), at        least one of the compared camera pixels in either Rcptr_S1 or        Rcptr_S2 will not have received light. Since camera pixels        receiving light will be brighter than camera pixels not        receiving light, the operation MIN(Rcptr_S1, Rcptr_S2), provides        an image where only pixels in set L [i.e. ∈L] are lit, which is        a good approximation of Tj, i.e. the j^(th) column in matrix T.

This implies that if sets of adjacent projector pixels in projectorpixel array 27 are lit in columns and in rows (or any arbitrary pair ofpatterns that intersect at only one point, i.e. share only one projectorpixel j in common), and a first collection of captured images Rcptr_Syare made for the lit columns of projector pixels and a second collectionof captured images Rcptr_Sx are made for the lit rows of projectorpixels, then the light transport coefficients for any individualprojector pixel j may be obtained by comparing both collections andidentifying the region L where a captured image of a lit columnintersect a captured image of a lit row, the intersection correspondingto a light ray projected by activation of projector pixel j, alone.

Thus, a method of determining transport matrix T is to collect a firstset of images Rcptr_Sy_1 to Rcptr_Sy_q, corresponding to q capturedimages of q lit columns of projector pixels, and construct a second setof images Rcptr_Sx_1 to Rcptr_Sx_p corresponding to p captured images ofp lit rows of projector pixels. Then for all projector pixels j=1 toj=(p×q) within projector pixel array 27, there exists a pair of row andcolumn captured images, Rcptr_Sy_a and Rcptr_Sx_b, such that theintersection of Rcptr_Sy_a and Rcptr_Sx_b correspond to a light rayfootprint created by activation of the target project pixel under-test,j. Therefore, one needs to construct sets of projection images:

-   -   Rprjct_Sy_1 to Rprjct_Sy_q and Rprjct_Sx_1 to Rprjct_Sx_p        where each projection image Rprjct_Sy_1 to Rprjct_Sy_q is paired        with any of projection images Rprjct_Sx_1 to Rprjct_Sx_p such        that each pair of projection images shares only one light        footprint in common. That is,

∀j∈{1, . . . , (p×q)} ∃ Rprjct_(—) Sy _(—) a, Rprjct_(—) Sx _(—)b|Rprjct_(—) Sy _(—) a ∩Rprjct_(—) Sx _(—) b={j}

The above formula being interpreted to mean that for all projectorpixels j in {1 . . . (p×q)} there exist a pair of projection images,each having a differently constructed pattern such that the intersectionof the constructed patterns intersect at a single point (i.e. patternsection, or a single light footprint region) corresponding to a commonprojector pixel, j. A basic example of such pairs of constructedpatterns would be projected pairs of vertical light lines and horizontallight lines. In this case, the intersection of the captured image of avertical light line and the captured image of a horizontal light linewould include all the camera pixels i that correspond to a targetprojector pixel under-test, j, (i.e. all camera pixels i that lie withina light ray footprint created by a light ray emitted from project pixelunder-test j).

Therefore, any column Tj [where j=1 to (p×q)] within transport matrix Tcan be synthesized from images Rcptr_Sy_1 to Rcptr_Sy_q and Rcptr_Sx_1to Rcptr_Sx_p. A scheme that satisfies this property is to use pixelcoordinates: let Rprjct_Sx_j be a first projected image such that onlypixels with an x-coordinate equal to j are turned ON, and letRprjct_Sy_k be a second projected image such that only pixels with ay-coordinate equal to k are turned ON. Then MIN(Rprjct_Sx_j,Rprjct_Sy_k) gives an image corresponding to projector pixel (j, k)being turned ON. This process can be better understood with reference toFIGS. 5A, 5B, 6A, 6B, 7A, and 7B.

In FIG. 5A, a scene, or display environment is illustratively shown as aflat surface 41 with a checkerboard pattern. It is to be understood thatthe checkerboard pattern is shown purely to facilitate the presentdescription by providing a contrast for projected light lines, and thedisplay environment need not have any pattern and may be of irregularshape. The relative location of each vertical light line and horizontallight line are known since it is known which projector pixels wereturned in their creation, and their known relative displacement may beused to calibrate real projector 21, as is more fully explained below.

Firstly, a bright vertical line, or vertical light beam, 47 _(—) k(i.e., column of light rays emitted simultaneously from a column ofprojection pixels), is projected onto surface 41 by real projector 21.In the present case, bright vertical line 47 _(—) k is generated byturning ON all projector pixels within projector pixel array 27 thathave a y-coordinate equal to k. Real camera 25 then captures this realimage, Rcptr_Sy_k, as one example of a lit column of projector pixels.

In FIG. 5B, where all elements similar to those of FIG. 5A have similarreference characters, real projector 21 projects a second brightvertical line 47 _(—) t of light rays onto surface 41. In this case,bright vertical line 47 _(—) t is generated by turning ON all projectorpixels having a y-coordinate equal to t. Real camera 25 then capturesthis image, Rcptr_Sy_t, as another example of a lit column of projectorpixels. It is to be understood that real projector 21 could project aseparate bright vertical line of light rays for each of the q columns ofprojector pixel array 27, and real camera 25 could capture a separateimage of each projected bright vertical line.

With reference to FIG. 6A, all elements similar to those of FIGS. 5A and5B have similar reference characters and are described above. In thepresent case, real projector 21 preferably projects a bright horizontalline, i.e. horizontal light beam, 49 _(—) j made up of light raysemitted simultaneously from a row of projection pixels onto projectionsurface 41. Bright horizontal line 49 _(—) j may be generated by turningON all projector pixels having an x-coordinate equal to j. Real camera25 then captures this real image, Rcptr_Sx_j, as one example of a litrow of projector pixels.

In FIG. 6B, real projector 21 projects a second bright horizontal line49 _(—) r (made up of simultaneously lit, individual light rays) ontosurface 41. As before, bright horizontal line 49 _(—) r may be generatedby turning ON all projector pixels having an x-coordinate equal to r.Real camera 25 then captures this real image, Rcptr_Sx_r, as anotherexample of a lit row of projector pixels. It is to be understood thatreal projector 21 could project a separate horizontal line of light raysfor each of the p rows in projector pixel array 27, and real camera 25could capture a separate image of each projected bright horizontal lineof light rays.

If one now conducts a pixel-by-pixel comparison of captured imageRcptr_Sy_k from FIG. 5A and captured image Rcptr_Sx_j from FIG. 6A (oralternatively compares only their respective bright vertical and brighthorizontal lines), using operation MIN(Rcptr_Sx_j, Rcptr_Sy_k) to retainonly the darker of two compared image pixels and discarding the brighterof the two, one would generate an image 43′, as shown in FIG. 7A. Allelements in FIGS. 7A and 7B similar to those of FIGS. 5A, 5B, 6A, and 6Bhave similar reference characters with the addition of an apostrophe,and are described above.

Since most of image Rcptr_Sx_j (FIG. 5A) is the same as image Rcptr_Sy_k(i.e. they mostly consist of the checkerboard pattern on flat surface 41with projected light rays), retaining the darker of two compared pixelsdoes not change the majority of the resultant image. That is, if twocompared pixels are relatively the same, then electing either pixel overthe other does not much affect the resultant image 43′. However, when apixel on bright vertical line 47 _(—) k in captured image Rcptr_Sy_k iscompared with a corresponding pixel in captured image Rcptr_Sx_j thatdoes not lie on a bright horizontal line 49 _(—) j, then retaining thedarker of the two image pixels will discard the pixel on the brightvertical line from image Rcptr_Sy_k, and retain the plain pixel fromimage Rcptr_Sx_j, which is illuminated by ambient light and not by aprojector pixel. Therefore, bright vertical line 47 _(—) k is eliminatedfrom generated image 43′. Similarly, when a pixel on bright horizontalline 49 _(—) j in image Rcptr_Sx_j is compared with a correspondingpixel in image Rcptr_Sy_k that does not lie on bright vertical line 47_(—) k, then retaining the darker of the two pixels will discard thepixel on the bright horizontal line from image Rcptr_Sx_j, and retainthe plain pixel from image Rcptr_Sy_k, which is illuminated by ambientlight and not by a projector pixel. Consequently, horizontal line 49_(—) j is also eliminated from generated image 43′. However, within theregion where bright vertical line 47 _(—) k intersects bright horizontalline 49 _(—) j, both compared image pixels are brightly lit pixelsshowing an impact by a light ray. Comparison of these two image pixelswithin this intersection region will result in either of the two brightbeam pixels being selected for image 41′. As a result, image 41′ willshow a brightly lit region 53 corresponding to a projected light rayemitted from coordinates (j,k) of projector pixel array 27. Thus, thelight transport coefficients for the projector pixel having coordinates(j,k) can be extracted from generated image 53 without having tophysically capture an image of a light ray individually projected fromthe projector pixel at (j,k).

A second example is shown in FIG. 7B, where the combination of realcaptured images corresponding to FIGS. 5B and 6B (which wouldrespectively correspond to real captured images Rcptr_Sy_t andRcptr_Sx_r following the above-described naming convention), results ina second brightly lit region 55 corresponding to a projected light rayemitted from coordinates (r,t) of projector pixel array 27.

A similar process may be followed to identify the light transportcoefficients of every projector pixel j in projector pixel array 27without having to individually turn ON and project each projector pixelj, one-at-a-time. This method of generating an image of ahypothetically, singularly activated projector pixel to obtain theprojector pixel's light transport coefficients requires at most only(p+q) captured images, one for each row and column of projector pixelsin projector pixel array 27 of real projector 21. Furthermore, once allthe pixel projection locations have been identified, the (p+q) capturedimages may be discarded, and all that needs to be saved is an index andcorresponding footprint information.

An example of this approach is shown in FIG. 8, where a partial view ofprojector pixel array 27 (illustrated as a cross-hatch pattern) iscompared to a partial view of camera sensor pixel array 29 (alsoillustrated as a cross-hatch pattern). For illustrative purposes, thecross-hatch pattern illustrating the partial camera sensor pixel array29 is made denser than the cross-hatch pattern representing projectorpixel array 27 in order to better illustrate that, in the presentexample, pixel density (i.e. resolution) of real camera 25 is preferablygreater than the resolution of real projector 21, and thus a light rayemitted from a single projector pixel j may create a light footprint(such as F1) spanning several camera pixels i.

In the present example of FIG. 8, an index of real projector pixel array27 is represented as a partial array with circles 1, 2, 3, . . . (q+1) .. . (2q+1) . . . etc. representing individual projector pixels j.Similarly, real camera sensor pixel array 29 is shown superimposed by acorresponding partial array of circular light footprints F1, F2, F3, . .. F(q+1), . . . etc. representing the footprint informationcorresponding to individually activated projector pixels j (assuminglight scattering is ignored). Thus, footprints F1, F2, F3, . . . F(q+1),. . . etc. respectively correspond to projector pixels 1, 2, 3, . . .(q+1), etc.

Following this approach, only two sets of information need to be stored.A first set of information corresponds to an index of projector pixels jand a second set of information corresponds to footprint informationassociating groups of camera pixels i with each projector pixel j. Inother words, zero-valued coefficients need not be stored, which greatlyreduces the memory requirements.

A second example of organizing this information is shown in FIG. 9,where an index 61 of projector pixels j is shown to point to, orcorrespond to, group 63 of grayscale (i.e. non-zero valued, or “NZGrayscale”) camera pixel i information (i.e. corresponding to aresultant light ray footprint).

Having shown how to reduce the number of images that need to be capturedand stored to generate the needed light transport coefficients, and howto reduce the amount of data that needs to be stored and manipulated,for dual photography, the following next section addresses some of thepractical difficulties of implementing dual photography. As is explainedabove, a light transport matrix T can be very large, and its use (or theuse of its transpose, the dual light transport matrix T^(T)) requireslarge amounts of active memory (for example, DRAM) and excessivecomputational processing power/time. Therefore, general use of the dualimage has heretofore not been practical.

To more efficiently construct a dual image, one first notes that

Vcptr″=T^(T)Vprjct″

Since the virtual camera sensor pixel array 27″ in virtual camera 21″corresponds in actuality to real projector pixel array 27 of realprojector 21 (see FIGS. 2A and 2B), it is convenient to use the sameindex j to denote any virtual camera pixel within Vcptr″ obtained byvirtual camera 21″. Therefore, a relationship between each virtualprojector pixel j in a virtual captured image versus a corresponding rowof elements in T^(T) may be denoted as

Vcptr″(j)=T ^(T) _(j) Vprjct

where T^(T) _(j) refers to the j^(th) row in T^(T).

As is explained above, T^(T) is the matrix transpose of light transportmatrix T (i.e. matrix T turned on its diagonal), and the values of rowT^(T) _(j) (where j is any value from 1 to (p×q)) therefore correspondto the j^(th) column of matrix T (i.e. T_(COL) _(—) J). Since eachcolumn of T has (m×n) elements (i.e. equivalent to the pixel resolutionof real camera 25), this would appear to be a very large number ofelements. However, recalling that in the present implementation, only alimited number of elements in each column of matrix T are non-zero (i.e.only those corresponding to camera sensor pixels i upon which shone theintersection of a vertical and horizontal light beam, i.e. a lightfootprint), it is self apparent that only a few of the (m×n) elementswithin in each column j of matrix T (and subsequently in each row T^(T)_(j)) are non-zero. Therefore, it is not necessary to examine allelements in T_(COL) _(—) j when computing Vcptr″ (1). Indeed, as isexplained above in reference to FIGS. 8 and 9, it is preferred that onlya single index 61 showing all individually activated projector pixels j,and their corresponding light ray footprint information 63 be stored.

As is shown in FIG. 9, index 61 associates a listing of grayscaleentries 63 for each projected light ray (from an individual projectorpixel j of real projector 21). It is to be understood that each group ofgrayscale entries 63 corresponds to the non-zero entries within eachcolumn of T, and only these non-zero grayscale values need to beexamined during each matrix operation of a column of light transportmatrix T. Thus, the number of calculations needed to determine eachvalue of Vcptr″(j)=T^(T) _(j)Vprjct″ is greatly reduced.

In other words, this subset of elements, SVcptr″(G), within each columnG of matrix T that needs to be processed may be defined as T_(COL) _(—)G(a), where a is an index for any virtually captured pixel [i.e. a ∈ {1,. . . , (p×q)}] in Vcptr″(G). Therefore, for each SVcptr″(G), one candefine the set of elements to be examined as:

SVcptr″(G)={a|∀ z ∈ {1, . . . , (p×q)}T _(COL) _(—) G(a)≧T T _(COL) _(—)z(a)}

Since in general ∥SVcptr″(G)∥<<(p×q), it takes significantly less timeto compute:

${{Vcptr}^{''}(G)} = {\sum\limits_{\alpha \in {{SVcptr}^{''}{(G)}}}^{\;}{{T_{G}^{T}(\alpha)}{{Vprjct}^{''}(\alpha)}}}$

than to compute:

Vcptr″(j)=T ^(T) _(j) Vprjct″

An example of a dual image generated using this method is shown in FIGS.10A and 10B. FIG. 10A shows a primal image, as projected by a realprojector. FIG. 10B shows the resultant dual image computed by animplementation of the present method. The dual image of FIG. 10Brepresents the image virtually captured by the real projector 27 (i.e.virtual camera 21″), or stated differently, the image as “seen” by thereal projector 21.

The above discussion shows how to compute dual images efficiently from areduced set of images, which saves image capture time as well ascomputation time. As is explained above, the real captured images anddual captured images can be used to calibrate both real camera 25 andreal projector 21, respectively.

That is, since the images from the projector's view cannot be directlycaptured, a straightforward solution is to construct projector-viewimages (i.e. dual images) from corresponding camera images using dualphotography techniques, and then to calibrate the projector using theconstructed images. For example, after having taken several real imagesat different angles of a known object with carefully measured features,the real camera can be calibrated by using the known dimensions of theobject to compensate for distortions in the captured images arising fromthe different angle views. The virtual images, as seen by the realprojector, can then be generated from the same captured images usingdual photography techniques, as described above, and the real projectormay be calibrated in a manner analogous to the real camera.

A possible set-back associated with this straight forward method,however, is the difficulty in generating and manipulating the lighttransport matrix T, and operating on the large image vectors resultingfrom the large number of camera and projector image pixels. Althoughthis labor-intensive and expensive process is mitigated substantially byusing the dual photography method described above, for purposes ofcalibrating real projector 21 in a projector-camera system, such as thatshown in FIG. 2A, Applicants have developed a novel method that avoidsthe need for generating a full T matrix and for creating a full dualimage, while still taking advantage of some of the benefits of using adual image (i.e. an image as “seen” by real projector 21) to facilitatecalibration of real projector 21.

Assuming that a projection scene is relatively flat, then one cangreatly reduce the number of light footprints points for which a lighttransport coefficient needs to be determined. In other words, thegeneration of a full T matrix can be avoided altogether by noting thatto calibrate the projector, one does not need to construct an entiredual image, but only needs to determine the location of a strategic setof points, particularly if the projection surface is not very irregular.For example, the strategic set of points may be the corners of thesquares within the checker board pattern on flat surface 41 of FIG. 5A,as seen by the projector. The greater the irregularity of the projectionscene surface, the greater the number of points in the strategicallyselected set of points. Conversely, the flatter the projection scene,the fewer the number of points in the strategic set of points.

Applicants have adapted a homography-based method to achieve this goalof a reduced number of points in the strategic set, and thus avoid thegeneration and manipulation of a full T matrix, or a simplified T matrixdescribed above, and further avoid the full dual-image generationprocess by incorporating some features of dual-image generation into thecalibration process, itself. This alternate embodiment of the presentinvention directly computes the coordinates of the checker cornerfeatures (or other known feature) across the projector-view imageswithout requiring the construction of the dual images and the detectionof the corners from the constructed dual images.

In this novel method, one may use the real camera to capture images of aplanar checkerboard and detect the checker block corners across thecaptured images. It is to be understood that that a checkerboard isbeing used purely for illustrative purposes, and any scene may becaptured. In the present case, since the orientation of the projectedvertical and horizontal lines are known, variations in the projectedlines can be identified and corrected, irrespective of the scene. Thus,the projector-camera system need not be calibrated beforehand. Rather,all elements of the projector-camera system are calibrated at the sametime.

Secondly, it is observed that projector images follow the so-calledperspective projection model, which relates two (or more) views of asingle scene as seen by two (or more) separated sources. That is,different viewing sources will “see” a different view (or image) of thesame scene since the different sources are located at different anglesto the scene. However, since there is only one real scene (irrespectiveof the number of views of the scene), one can generate a mathematicalrelationship between the different views that will associate any pointon any one view to a corresponding point on the real scene (and therebyto a corresponding point in each of the other views).

If one of these separate views is assumed to be a virtual image as“seen” by a real projector, while a second separate view is deemed to bea real image captured by a real camera, then the perspective projectionmodel (which relates the two views to the common, real scene) wouldpermit one to extract from the captured real image some informationrelating to the virtual image, without generating a full dual image.

Using this approach, Applicants have devised a method of extractingsufficient information for calibrating a real projector withoutrequiring a full dual image. Thus, although the dual image is not fullycreated, one can still apply a camera calibration technique to aprojector, albeit in a roundabout way.

Under the perspective projection model, the relationship between twoimage projections of a planar object from different views is a simplelinear projective transformation or homography. This transformationrelates the coordinates of any point on the planar object (i.e. ahomogeneous coordinate) to the coordinates of a corresponding point on aspecific view of the planar object. In the present embodiment, theprojector-view image of the planar checkerboard is a homography of thecorresponding camera image. Specifically, for any point P on the realscene (i.e. checkerboard), its homogeneous coordinate in theprojector-view image Up=(up, up, 1) and the coordinate in the cameraimage Uc=(uc, uc, 1) satisfy the following equation,

Up=λHUc

where λ is a scalar and H is a 3×3 homography transformation matrix (asis known in the art) of which the bottom right entry is set to 1. Thepair of corresponding coordinates provides 3 linear equations, where oneof the equations determines the scalar and the other two are used todetermine H, the homography transformation matrix. Since there are 8unknown entries in 3×3 matrix H, given the correspondence between Ncoordinate points (where N≧4) on the checker board, the homographybetween the projector-view image and the camera image can be recoveredby solving the 2N linear equations. The greater the number of N, thelower the error relating coordinate points between the projector-viewand the camera image.

To obtain the corresponding coordinates, 10 white points are preferablyprojected on a scene, such as the checkerboard pattern. An image of thecheckerboard with the projected white points is then captured using areal camera, such as real camera 25, and the coordinates of the 10points in the camera image are computed. In the present process, it isonly necessary that the ten points be distinguished during thecomputation of their corresponding coordinates in the captured image.This may be achieved by projecting the ten points sequentially, anddetermining their corresponding coordinates, in turn. Alternatively,differently colored points may be projected simultaneously, and thedifferent points identified by color.

Since the projector preferably projected the points in a known relationto each other, the coordinates of the points in the projected image areknown. This results in 10 pairs of corresponding coordinates, one set ascaptured by the real camera and a second set as projected by the realprojector. Once the homography is recovered, the coordinates of thecheckerboard corners detected in the camera images can be directlytransformed to compute the corresponding corner coordinates in theprojector-view images. The projector parameters can then be calibratedusing a camera calibration method, such as the one described above.

An example of this approach is shown in FIG. 11. In FIG. 11, the featurecapture results are as follows. The circles, or dots, not shown inoutline (for example dots 81) were used for estimating nomography, whilethe outlined circles, or dots, (for example dots 83) are the cornerpoint features. As can be seen, the outlined dots 83 are on the actualcorners, indicating that the projector coordinates for each detectedcorner has been correctly captured.

It is to be understood, however, that instead of projecting dots, onemay use the method described above of projecting multiple pairs of linesthat intersect at one point (i.e. horizontal and vertical lines asdescribed above), and extracting dot information from the oneintersection point of each pair of lines. Furthermore, it is notnecessary to project the lines onto a known pattern, such as thecheckerboard pattern described above, since the spatial relationshipbetween the pairs of projected lines are already known. That is, sinceone knows which columns and rows of the projector pixel array are turnedon simultaneously, one knows their spatial relation. Furthermore, it isto be noted that the camera need not be pre-calibrated to the particularscene. Rather, the projector and camera may be calibrated at the sametime.

As is explained above, this approach of using patterns to extract dotinformation includes a series of pattern projection and image capturesteps. Thus, when an uncalibrated camera captures each projected pattern(i.e. vertical or horizontal line), any distortion of the projectedpattern due to irregularities on the projection surface may be ignoredfor the moment. When all the patterns have been captured, the spatialrelation (or transformation) between the projector and camera can bemade since the spatial relation between the vertical and horizontallines are known. That is, the spatial relation between the vertical andhorizontal lines, as projected, are known since one knows whichprojector pixels were turned on during their generation. Furthermore,since one knows the true orientation of the vertical and horizontallines, one can compensate for surface irregularities on the projectionscene and view angles.

When a captured image of a vertical line is combined with a capturedimage of a horizontal line to extract dot information, one knows whatprojector pixel relates to the extracted dot, and one further candetermine what group of camera pixels correspond to the extracted dot(as determined by the intersection of the vertical and horizontallines). Thus, the homography relationship between the projector andcamera (i.e. the transformation of one view for the other) can beachieved without need of calibrating either device individually to anyspecific projection scene (or projection surface).

This approach borrows from the above-described, simplified method forgenerating the transport matrix, T, but reduces the number of neededimage projection-and-capture steps from (p+q) to a fraction of (p+q)determined by the number of desired dots. Although as little as 4 dotsmay be used to calibrate a projector-camera system whose sceneenvironment is a flat projection surface, in order to account for someirregularities in a projection surface, for possible errors in the imagecapture steps, and errors in identifying projected light lines orpatterns due to existence of ambient light noise, it has been found thatprojecting seven horizontal light lines and seven vertical light linesto generate as many as fourteen to forty-nine dots is sufficient forovercoming most errors. It is to be understood that the greater theamount of surface irregularities in the projection scene, the greaterthe number of needed intersecting pairs of lines, but this number ofseven vertical lines and seven horizontal lines has been found to besufficient for many real-world situations. A reason why much fewer than(p+q) dots are needed is because one is not trying to generate a fulltransport matrix for use in dual photography, but rather, one isinterested in identifying a few known relational dots for purposes offinding the nomography relationship between two views, i.e. between thereal view of the real camera and the virtual view from the realprojector. However, it should be emphasized that an estimate of a fulllight transport matrix T can be generated from the few relational dotsby assuming a smooth transition between the dots. That is, once thehomography relationship is obtained, the same relationship of Up=λHUccan be used to fill-in gaps in construction of an estimated lighttransport matrix T.

A still alternate application of the above described method forgenerating transport matrix T can be applied toward solving thefollowing problem: given a projector and camera pair, if one would likethe camera “to see” a desired view on a given scene (i.e. projectionenvironment), what should the projector illuminate onto the scene inorder to produce that desired view? This task is termed “ViewProjection” hereinafter.

Clearly, finding the needed image to be projected requires detailed andprecise knowledge about the geometry of the scene and the projectioncharacteristics of the projector device, along with the photometricproperties of the scene so that the reflection of light from theprojector produces the exact desired view. For display surfaces withuniform albedo, i.e. reflectivity, and parametric shapes, thesenecessary measurements can be made explicitly.

Producing high quality imagery on arbitrarily complex nonparametricshapes with non-uniform surface reflectance properties, however, isstill a daunting task requiring familiarity with a wide variety of toolsand much human intervention. A goal of the present invention is todevise a measurement process that is completely automatic and requiresno user intervention or parameter tweaking beyond casual hardware setup.In other words, the projector-camera system that achieves ViewProjection capabilities should completely calibrate itself with a singletouch of a button.

A projection system that is capable of displaying correctly on complexsurfaces under challenging settings has many real world applications.Making it fully automatic further allows it to be deployed in settingswhere professional display setup is not always available. Another goalof the present invention is to reduce the requirement on exotic imagingequipment, so that high quality calibration can be achieved withoff-the-shelf components within an average consumer's reach. This goalmay be achieved by combining some of the features and benefits of theprevious embodiments in an automated and simplified process.

As is explained above in reference to the image setup of FIG. 2A, whichconsists of one real projector 21 and one real camera 25, real projector21 has a p-row-by-q-column projector pixel array 27 while the realcamera 25 has an m-row-by-n-column camera sensor pixel array 29. Lightrays emitting from real projector 21 bounce off a scene (i.e. projectionsurface) and some of them eventually reach real camera sensor pixelarray 29. In general, each ray of light is dispersed, reflected, andrefracted in the scene and hits camera sensor pixel array 29 at a numberof different locations. Thus, the light ray emitted from projector pixelj reaches real camera 25 and forms an m-by-n image across the entiretyof camera sensor pixel array 29, where each camera pixel i receives acertain amount of light. If the image projected is represented as a(p×q)-element real image vector Rprjct, and the image captured isrepresented as an (m×n)-element captured image vector Rcptr, then thelight transport between real projector 21 and real camera 25 can bewritten as

Rcptr=T Rprjct

where T is the light transport matrix. As is illustrated in FIG. 2B, ithas been shown that by the principle of Helmholtz reciprocity, theequation

Vcptr″=T^(T)Vprjct″

can be used to model a “dual” setup where projector 21 is viewed as ap-row-by-q-column virtual camera 21″, and real camera 25 is viewed as anm-row-by-n-column virtual projector 25″. This enables “DualPhotography”, where one can synthesize images that appear as though theywere captured by real projector 21, with the scene appearing as ifilluminated by light emanating from real camera 25.

View Projection, the object of the present embodiment, addresses adifferent problem. In View Projection, one is interested in finding aprojector image that, when used to illuminate a scene, allows the camerato capture a predefined, desired image. In light transport terms, one isprovided with T and with a desired Rcptr, and one wants to recover, i.e.generate, Rprjct. Clearly, Rprjct can be found by the followingrelation:

Rprjct=T ⁻¹ Rcptr

if one can determine the inverse, T⁻¹, of the light transport matrix T.

The inverse of light transport matrix T has been used in the past forother purposes. For example, in article “A Theory of Inverse LightTransport” by Seitz et al. (IEEE International Conference onCommunications, ICC V05), hereby incorporated in its entirety byreference, the inverse of the light transport matrix T is used toanalyze the way light bounces in arbitrary scenes. In this approach, ascene is decomposed into a sum of η-bounce images, where each imagerecords the contribution of light that bounces η times before reaching acamera. Using a matrix of “impulse images”, each η-bounce image iscomputed to infer how light propagates through the scene.

The problem being addressed by Seitz et al., of course, is differentthan that posed by View Projection, where one is provided with, orenvisions, a desired image (Rcptr, for example) and wants to infer anunknown projection source that can produce the desired image. In thiscase, it would be beneficial to compute Rprjct=T⁻¹Rcptr. Note that T^(T)Vprjct″ of Dual Photography renders a virtually captured image Vcptr″,which is a different view of Rcptr, but does not show the real source ofprojected image Rprjct.

The inverse, T⁻¹, of transport matrix T is however more difficult tocompute than its transpose, T^(T), requiring much more computationalresources. Indeed, the sheer size of T makes computing T⁻¹ an extremelychallenging task requiring tremendous computational resources. Worse, itis not always possible to find the inverse of an arbitrary matrix. Thatis, some matrixes may not have an inverse.

As it is known in the art, the identity matrix, or unit matrix, isdenoted by I, and has the property that for a given matrix A, thefollowing relationship holds:

AI=IA=A

If matrix A were a matrix of order m by n, then the pre-multiplicativeidentity matrix I would be of order m by m, while thepost-multiplicative identity matrix I would be of order n by n.

The multiplicative inverse of a matrix is typically defined in terms ofidentify matrix I. A left multiplicative inverse of a matrix A is amatrix B such that BA=I, and a right multiplicative inverse of a matrixA is a matrix C such that AC=I If the left and right multiplicativeinverse of a matrix A are equal, then the left and right multiplicativeinverse are simply called the “multiplicative inverse”, or “inverse”,and is denoted by A⁻¹.

Fortunately, Applicants have found that in many display settings anapproximation to the inverse T⁻¹ of light transport matrix T, which maybe computed more simply, may suffice. Creating suitable approximation tothe inverse of a matrix, however, requires meeting certain criteria notgenerally present in light transport matrix T. Before creating theapproximation to the inverse T⁻¹, one therefore needs to create asuitably modified light transport matrix T.

In general, T⁻¹≠T^(T). However, one constraint used above in thegeneration of an estimation of T^(T) is likewise useful in theconstruction of T⁻¹. As is discussed above, for most projector-cameradisplay applications, one may generally use the limitation that any twodistinct light rays j and k emitted from a projector will hit a camerasensor at distinct parts, i.e., there is usually little overlap in thecamera sensor pixels hit by light from each of the light rays j and k.This characteristic is termed the “Display Constraint” hereinafter.

In general, the Display Constraint may be violated if there issignificant light scatter in the scene, such as the example given abovewhere the scene consists of a glass of milk, and the light rays arediffused by the milk resulting in significant overlap. Another exampleof a scene that violates the Display Constraint is shown in FIG. 12,which includes two wine glasses between a projector and a projectionscene, resulting in significant light scattering. In this example, aprojected image consisting of lettering is projected onto the scene andshown to experience much optical distortion due to the wine glasses.Below is provided a method for compensating for such light scatteringwhen the Display Constraint is violated, but for the present discussion,it is first assumed that the Display Constraint is met and each pixelprojected will be distinct from the next.

As is also explained above, each column of the transport matrix T is theprojection image resulting from one pixel from the projector. Thus, allof the column entries have zero values except those corresponding to thecamera pixels hit by light emitting from the corresponding projectorpixel.

Applicants noted that if adjacent light footprints (created byadjacently lit projector pixels) have no overlap, then columns of Twould be orthogonal. In real-world applications, by the DisplayConstraint, overlap between adjacent light footprints is minimal, whichimplies that light from different projector pixels will mostly hitdifferent camera pixels. As a result, if two columns of T are placednext to each other, the non-zero entries will not line up most of thetime, and their dot product will be close to zero. This implies that thecolumns of T are approximately orthogonal to each other. In the presenttechnique, these minimal overlapping pixels are removed from thecreation of the T matrix, which results in an approximate T matrix thatis orthogonal, by definition. As is explained more fully below, thistechnique permits further simplification of creation and manipulation ofthe T matrix.

By the Display Constraint, light from different projector pixels willmostly hit different camera pixels, i.e. camera pixels corresponding todifferent projector pixels will have little overlap with each other. Anexample of this is shown in FIG. 13, wherein a first light footprint,Projection_Pixel_1, from a first projector pixel shares only twoperimeter camera pixels 2 i with an adjacent second light footprint,Projector_Pixel_2, from a second projector pixel. This means that ifcolumns of light transport matrix T are placed next to each other, thenon-zero entries among the different columns will not line up most ofthe time (i.e. only the dimmer non-zero entries will have some overlap),and their dot product will be close to zero.

An example of this is shown in FIG. 14, where four columns C1 to C4 of alight transport matrix T are shown adjacent each other. In column C1,circles Pixel_1 c and Pixel_1 d identify the brightest parts (i.e. thecenter parts) of a first pixel's light footprint that is recorded as animage within column C1. As shown, the full light footprint includesother less bright pixel groups Pixel_1 b and Pixel_1 e. However, forease of explanation, only the brightest parts of each light footprintwithin each column C1-C4 is identified by circles, since the circlesidentify those pixel coefficient entries that will be retained inconstruction of a modified (i.e. estimated) light transport matrix T. Aswas shown in FIG. 13, the brightest part of any one light footprint doesnot overlap with the brightest part of any other light footprint. Thus,if the dimmer, perimeter sections of a light footprint (i.e. Pixel_1 band Pixel_1 e) are ignored to form a resized light footprint, then itcan be assured that no overlap will exist between the resized lightfootprints. As is shown in FIG. 14, if one compares adjacent circledgroups of pixels along adjacent columns, one will note that none of thecircled pixels in adjacent columns overlap. For example, partial lightfootprint Pixel_1 c in column C1 is offset in the horizontal directionfrom partial light footprint Pixel_2 c in column C2, which in turn isoffset from partial light footprint Pixel_3 c in column C3, which islikewise offset from partial light footprint Pixel_4 c in column C4, andso on. Similarly, none of the other bright partial light footprintcircled sections (i.e. Pixel_1 d, Pixel_2 b, Pixel_3 b, and Pixel_4 b)within each column line up with each other in the horizontal direction.

Since by the Display Constraint light from different projector pixelswill mostly hit different camera pixels, most column entries havezero-value entries, except those corresponding to a light footprintdefining camera pixels hit by light emitted from a correspondingprojector pixel. This means that if two columns of T are placed next toeach other, the non-zero entries will not line up most of the time, andtheir dot product will be close to zero. This implies that the columnsof T are orthogonal to each other.

This is particularly true if the dim perimeter parts of a lightfootprint are ignored, and one works with the resized light footprintsconsisting of only the brightest sections of a light footprint. In thiscase, columns of T become truly orthogonal to each other, meaning thatthat the transpose of a specific column multiplied by any column otherthan itself will produce a zero result.

Thus, the orthogonal nature of T may be assured by the way in which T isconstructed in the above-described process for generating a modified T.As is explained above, the amount of memory necessary for storing T islikewise reduced by storing only nonzero values within each column. Inthe modified T, this process is simplified because the nonzero values tobe stored can be quickly identified by storing only the brightest valuewithin each row of T, which automatically creates resized lightfootprints. For example in FIG. 14, one may identify the brightestpixels along rows of T (i.e. among adjacent image informationcorresponding to separate projector pixels and arranged in adjacentcolumns of T) as indicated by circles Pixel_1 c to Pixel_4 c, Pixel_2 bto Pixel_4 b and Pixel_1 d. If only the brightest valued entry withineach row is retained, one obtains the structure of FIG. 15. Theidentified bright sections may then be combined into a vectorrepresentation 40 of modified matrix T (hereinafter light transportmatrix T and modified light transport matrix T are used interchangeably,unless otherwise stated). As shown, vector representation 40 removes anyoverlap between adjacent columns of T by limiting itself to only thebrightest pixel values within each row of T, and effectively imposes theDisplay Constraint on an arbitrary scene. Consequently, this constructof T is orthogonal, by design. It is to be understood that to fullydefine modified light transport matrix T, one only needs a second matrixcolumn, or second array, to hold index information indicating whichgroups of partial light footprint sections belong to the same resizedlight footprint. In the present case, for example, groups Pixel_1 c andPixel_1 d are part of a single resized light footprint corresponding toa first pixel. Similarly, light footprint sections Pixel_2 b and Pixel_2c together form a second resized light footprint for a second pixel, andso on.

To determine an approximation to the inverse of T (i.e. to determineT⁻¹), it is beneficial to first note that AA⁻¹=I, and that the identitymatrix I is comprised of a matrix having entry values set to a value of“one” (i.e. 1) along a diagonal from its top left corner (starting atmatrix location (1,1)) to its bottom right corner (ending at matrixlocation (r,g)), and having entry values set to “zero” (i.e. 0)everywhere else. In order to compute T⁻¹, one first defines a matrix{hacek over (T)} such that each column in {hacek over (T)} is comprisedof normalized values, with each column in {hacek over (T)} correspondingto a column in light transport matrix T (or in modified light transportmatrix T). That is,

{hacek over (T)}r=Tr/(∥Tr∥)² , r=1, 2, 3, . . . , pq

where {hacek over (T)}r is the r^(th) column of {hacek over (T)}. Sincematrix operation ∥Tr∥ defines the square root of the sum of the squaresof all values in column r of matrix T, the square of ∥Tr∥ is simply thesum of the squares of all the values in column r. That is,

$\left( {{Tr}} \right)^{2} = \left\{ {\sum\limits_{ɛ = 1}^{ɛ = g}\left( {Tr}_{ɛ} \right)^{2}} \right\}$

By dividing each value entry in column r by the sum of the squares ofall the values entries in column r, operation {Tr/(∥Tr∥)²} has theeffect of normalizing the value entries in column r of matrix T. If onenow takes the transpose of {hacek over (T)}r, i.e. flips it on its sidesuch that the first column becomes the top row and the last columnbecomes the bottom row, the result will be rows of elements that are thenormalized values of corresponding columns of elements in T. Therefore,for every column in T, one has the following result:

({hacek over (T)}r ^(T))×(Tr)=1

and

({hacek over (T)}r ^(T))×(Tω)=0, for r≠ω

In other words, multiplying a column of T with a corresponding row in{hacek over (T)}r^(T) always results in numeral 1, and as one multipliesall the columns in T with the corresponding row in {hacek over(T)}r^(T), one produces a matrix with numeral 1's along its diagonal,and one may place zeroes everywhere else to fully populate the producedmatrix.

Therefore, in the case of matrix T, where columns are (or are made)orthogonal to each other, and given the specific construction of matrix{hacek over (T)}, it has been shown that the transpose of {hacek over(T)} is equivalent to the inverse of T (i.e. {hacek over (T)}^(T)=T⁻¹),by definition, or at least {hacek over (T)} is a left multiplicativeinverse of T. Therefore, Rprjct={hacek over (T)}^(T) Rcptr.

Note that only the part of the projector pixels that actually hit thecamera sensor can be recovered. For the projector pixels not hitting anyof the camera pixels, the corresponding columns in T contain zeros andthe above equation of {hacek over (T)}r=Tr/(∥Tr∥)² is undefined. In suchcases it is preferred that the corresponding columns in {hacek over (T)}be set as zero columns. Thus, {hacek over (T)}^(T) is the inverse of thepart of T that covers the overlapping area of the field-of-views of theprojector and the camera. It only recovers the projector pixels inRprjct that fall in the overlapping area and blacks out the otherpixels.

In the following discussion, matrix {hacek over (T)}^(T) is called theView Projection matrix, such that given a desired view c, one can findan image p defined as p=({hacek over (T)}^(T) c) such that projectingimage p produces a scene which, when viewed from the reference cameralocation, has the same appearance as c. Since {hacek over (T)}^(T) iseffectively an approximation of inverse matrix T⁻¹ and is used as suchherein, for the rest of the following discussion inverse matrix T⁻¹ andView Projection matrix {hacek over (T)}^(T) may be used interchangeably,unless otherwise indicated.

Although an efficient method for computing an approximation of trueinverse matrix T⁻¹ has been demonstrated, in its raw form, it is still astaggeringly large matrix, requiring large amounts of storage and ofcourse significant computation power just to perform matrixmultiplication. Again, approximation of the true inverse matrix T⁻¹ issubject to the Display Constraint.

Therefore, it is advisable to apply to View Projection matrix {hacekover (T)}^(T) a similar compacting technique as is described above forreducing T^(T). In this manner, {hacek over (T)}^(T) may be reduced to arepresentation consisting of a first column of nonzero entries and asecond column of corresponding index values. Recall that the DisplayConstraint dictates that the non-zero entries in the distinct columns ofT do not line up on the same row. This means that on each row of T,there will only be one non-zero entry (or one entry above a predefinedthreshold value) and one need only store that entry along with itscolumn index in order to represent the entire matrix T (with theunderstanding that any non-stored T entry location is designated with azero value, by default). Consequently, the space needed to store T canbe reduced down to that of a single column of nonzero T entries plus asecond column of index values.

In a real world setting, however, many of the unlit entries of T (i.e.those camera pixels i not within a light footprint) will not be exactlyzero due to sensor noise (i.e. light picked up by a camera pixel, butwhich does not originate from a singularly designated, i.e. turned ON,projector pixel). Thus, there may be multiple non-zero values in eachrow, with one non-zero value entry corresponding to a light footprintand the remaining non-zero entries being due to light noise. In suchcases, it is likely that the one non-zero entry corresponding to a lightfootprint is will have the highest value in the row (i.e. have thebrightest value). Therefore in order to filter out light noise, themaximum valued entry in each row (i.e. the row-maximum value) may beidentified and designated as the one non-zero entry for that row thatcorresponds to a light footprint. This eliminates any low valued entriesdue to light noise. By identifying these row-maximum entry values, oneobtains the non-zero values of each column. Therefore, for eachprojector pixel, one identifies the corresponding set of camera pixels,along with the distribution of light intensity among these camerapixels. Typically, this set of corresponding camera pixels (associatedto any given projector pixel) is a very small subset of the entirecamera image. Since only the entries from these corresponding camerapixels need to be considered during matrix operations, performing viewprojection image transformation using this sparse matrix representationis very efficient.

In practice, it is preferred to use a fixed pattern scanning techniquewhich exploits the Display Constraint to speed up the process. As isexplained above, the contribution from a projector pixel j maps tocolumn j of the light transport matrix T. It follows that a set ofprojector pixels S1 □ {1, . . . , p×q} maps to a corresponding set ofcolumns in T. Considering two such sets of pixels S1 and S2 where S1 ∩S2={j}, let the two images captured when turning on the two sets ofpixels be C_(S1) and C_(S2). It can be shown that

Tj≈MIN(C_(S1), C_(S2))

where Tj is the j^(th) column of T. C_(S1), C_(S2) are the sum of therespective columns of T, i.e. Cs=Σ_(j∈S) Tj. Given that the contributionof each individual projector pixel j is mapped to distinct parts of thecamera sensor, there is a common set of pixels l □ {1, . . . ,m×n} inthe captured images C_(S1), C_(S2) that corresponds to projector pixelj. Now, as S1 ∩ S2={j}, for the rest of the captured image pixels ∉ l,at least one of the images would not have received light from one of theprojector pixel sets, S1 or S2. Since pixels receiving light will bebrighter than pixels not receiving light, MIN(C_(S1), C_(S2)) renders animage where only pixels ∈ l are lit, which is a good approximation ofTj.

This implies that if one can construct sets

-   -   Y1, . . . , Yp, X1, . . . ,Xq,        where

∀i ∈ {1, . . . , (p×q)}, ∃X _(j) , Y _(k) |Xj ∩ Y _(k) ={i},

one can synthesize Tj where j=1, . . . , p×q from images C_(X1), . . . ,C_(Xq) and C_(Y1), . . . , C_(Yp).

In the above discussion, each pair of projected images has only oneintersection point in common. One construction that satisfies thisproperty uses the following pixel coordinates as indicators: let Xj bean image creating by turning on only those pixels whose x-coordinate isequal to j, and Yk be an image created by turning only those pixelswhose y-coordinate is equal to k. Then MIN(Xj, Yk) gives an image withcoordinates (j, k) turned ON. Using this method, capturing p+q imagesallows one to synthesize all p×q columns of T. It should be noted thatwhile this method for capturing T is faster than capturing each pixelindividually, and simpler than the adaptive scheme described in thereference to Sen et al., 2005, discussed above, this method may still beslower than other schemes previously proposed in the literature. Inparticular, a method for establishing projector-camera pixelcorrespondence is to project the X and Y coordinates in time sequentialbinary codes, in theory allowing all correspondence to be captured inlog(N) time, where N is the maximum coordinate value. In practice, thismethod places stringent requirements on the camera sensor resolution,especially when the least significant bits are being projected. One mayalso propose to project multiple stripes simultaneously to cut down thenumber of images needed. This however requires some way to identify eachof the stripes captured in an image, e.g. specifying a subset oflandmark stripes or exploiting color coding or other distinguishingpatterns. These schemes may indeed speed up the capture process, howeverthey introduce additional complexity and brittleness, and often oneneeds to tweak the feature detection parameters so the schemes workcorrectly. In the presently preferred embodiment, there are noparameters to be tweaked and unless there is a hardware failure, aone-time capture typically yields a good copy of a view projectionmatrix.

As is explained above, the view projection matrix, {hacek over (T)}hu T,is capable of compensating for geometric and photometric distortionsintroduced by projection optics, display surfaces, and their relativepositioning. The efficacy of the view projection capabilities of {hacekover (T)}^(T) was tested by means of a number of experiments.

To illustrate that one can control the desired appearance of a scene, itwas shown that one can dynamically change the appearance of a printedposter. An example is shown in FIG. 16. The Poster image on the left isshown under white light. The poster image on the right is shown underview projection illumination. The faces of the cubes are made to appearto have the same color. A sheet of white paper placed over the posterimage on the right reveals the actual projected image used to producethis desired view. It was found that viewers quickly lose track of theoriginal appearance of the poster after seeing the animation. While itis difficult to show in a paper, this scene as observed by the camera isquite close to that observed by the human eye. The poster was animatedby cycling the colors in the poster and making the colors appearuniform. Movies were also shown over the poster while photometriccompensation took place in real time to eliminate all traces of theunderlying poster image.

As mentioned earlier, the presently preferred construct of transportmatrix T depends on the Display Constraint, which stipulates minimaloverlap between adjacent light footprints resulting from adjacently litprojector pixels. An example of a situation where the Display Constraintis not upheld is discussed above in reference to FIG. 12, where two wineglasses are placed between a projector and a projection surface, orscene. As is further explained above, however, the view projectionmatrix {hacek over (T)}^(T) can enforce (i.e. impose) the DisplayConstraint on an arbitrary scene by selecting only the brightest pixelwithin each row of T in the creation of an approximation of the inverseof T. In order to test the efficacy of the algorithms of the presentinvention under various conditions (not just only under the DisplayConstraint, from which they derive their efficiency and justification)an experiment was conducted to determine how they behave when theDisplay Constraint is not satisfied. The experiment illustrated in FIG.12 was set up with the pair of wine glasses placed between the projectorand its display surface. Due to the significant amount of distortionsintroduced by the glassware, there is much overlap between the sensorlight footprints of distinct projector pixels. To this scene was thenapplied the same acquisition method and same algorithms for enforcingthe Display Constraint in the generation of the view projection matrix{hacek over (T)}^(T). It was found that the computed view projectionmatrix {hacek over (T)}^(T) was able to remove most of the unwanteddistortion and allow the projector to display a corrected image on thedesignated display surface, as shown in FIG. 17.

Stated differently, when the Display Constraint cannot be guaranteed,the method of generating view projection matrix {hacek over (T)}^(T),which forces a pseudo Display Constraint on a projection environment,can be applied to the generation of the light transport matrix T.Specifically, when generating the simplified light transport matrix T,as described above in reference to FIGS. 13-15, one may select thebrightest pixel in each row to identify components of a light footprint,whereby light diffusion error and light noise error can be greatlyreduced or eliminated.

Having defined a method for approximating the inverse of matrix T, itwill now be shown how dual photography can be used with an immersivedisplay system to achieve advanced and complex setups.

With reference to FIG. 18, in a preferred embodiment, a conventionalfront projector P1 (similar to real projector 21 of FIG. 2A) is used inconjunction with an immersive projector P2. As indicated byfield-of-view (i.e. FOV) 91, in the presently preferred embodiment, theportion of a display surface covered by the FOV 91 of front projector P1is a subset of the field-of-view of immersive projector P2, as indicatedby field-of-view lines 93. Since FOV 91 of front projector P1 isentirely within the scope of FOV 93 of immersive projector P2, it wouldbe desirable to have immersive projector P2 simulate the projected imageproduced by front projector P1. However, it should be emphasized that ingeneral FOV 91 of front projector P1 does not necessarily need tooverlap any part of FOV 93 of immersive projector P2.

Although it is not necessary for FOV 93 of immersive projector P2 tooverlap part of FOV 91 of front projector P1, it is desirable that twolight transport matrices separately associating a camera C to frontprojector P1 and to immersive projector P2 be created. As it would beunderstood, the two transport matrices may be generated separately sincethe FOV's of P1 and P2 do not necessarily overlap.

However, in the specific example of the presently preferred embodiment,camera C is placed such that the FOV 95 of camera C is a superset of FOV91 of front projector P1 and a subset of FOV 93 of immersive projectorP2. As indicated by FOV lines 95, the field-of-vision of camera Ccompletely encompasses FOV 91 of front projector P1, but is entirelyengrossed by FOV 93 of immersive projector P2. To simulate a projectedimage from front projector P1 using immersive projector P2, one firstdetermines a first light transport matrix, T₁, relating a firstprojected image p₁ from front projector P1 to a first captured image c₁captured by camera C such that c₁=T₁p₁. One then determines a secondlight transport matrix, T₂, relating a second projected image p₂ fromimmersive projector P2 to a second captured image c₂ captured by cameraC such that c₂=T₂p₂. Consequently, one has the following relation

c₁=T₁p₁

and

c₂=T₂p₂

In order to simulate projected image p₁ from front projector P1 usingimmersive projector P2, one needs c₁ (i.e. the captured, projected imagefrom front projector P1) to be the same as c₂ (i.e. the captured,projected image from immersive projector P2), i.e. one needs

c₂=c₁

which lead to the relation:

T₂p₂=T₁p₁

solving for p₂ (i.e. the image projected by immersive projector P2), oneobtains the following relation:

p ₂=(T ₂ ⁻¹)(T ₁ p ₁)

This means that to create image p₁, one can project the image directlyusing front projector P1, or the same effect can be achieved byprojecting a transformed image [defined as (T₂ ⁻¹)(T₁p₁)] on immersiveprojector P2. Note that the view projection matrices naturally conveythe projector-camera correspondence into projector-projectorcorrespondence.

Such a projection is shown in FIG. 19, where immersive projector P2 isused to simulate a front projector, such as projector P1 of FIG. 18. InFIG. 19, a virtual projector P1″, as simulated by immersive projectorP2, is denoted by dotted lines. Therefore, image p₁, as projected byfront projector P1 of FIG. 18, can be recreated by projectingtransformed image (T₂ ⁻¹)×(T₁p₁) on immersive projector P2 of FIG. 19.By doing so, viewers 100 a, 100 b, and 100 c do not have to concernthemselves with occluding any front projector, i.e. P1 or P1″. Clearly,the image is distortion free and movie playback on the virtual projectorcan be run in real time. It is to be understood that T₂ ⁻¹ can bereplaced by an approximation matrix {hacek over (T)}^(T) ₂, as explainedabove. As stated before, view projection matrix {hacek over (T)}^(T),which approximates an inverse matrix T⁻¹, can be freely substituted forT⁻¹ in the following discussions, unless otherwise stated.

An example of an image generated using this virtual projectorimplementation is shown in FIG. 20. A front projected image 101 issimulated using a large field-of-view display system. Projector 103,located along the bottom of FIG. 20 is part of the large field-of-viewdisplay system, and is used to generate image 101 shown in the center ofFIG. 20.

FIGS. 21A to 21C illustrate the quality of the simulation by showing areal front-projected image (FIG. 21A) and a simulated front-projectedimage (FIG. 21B) seamlessly coupled together side-by-side (FIG. 21C).FIG. 21A shows the right side of a front-projected image projected by areal front projector, such as P1 of FIG. 18. FIG. 21B shows thecorresponding left side of the front-projected image of FIG. 21A, but inFIG. 21B the left side of the front-projected image is projected by animmersive projector, such as P2 of FIG. 19, to simulate a virtual frontprojector, such as P1″ of FIG. 19. The quality of the simulated, leftside, front-projected image created by the immersive projector is betterillustrated in FIG. 21C, where the right side front-projected image ofFIG. 21A is shown joined to the left side front-projected image of FIG.21B, side-by-side, resulting in a seamless registration of the rightside and left side images created by a real front projector and asimulated, virtual front projector, respectively.

Two additional examples showing side-by-side comparisons of realfront-projected images created by a real front projector and simulatedfront-projected images created by an immersive projector are shown inFIGS. 22A and 22B. In both FIG. 22A and 22B, the left half of the shownimage is created by an immersive projector to simulate a display from avirtual front projector, and the right side half of the shown image iscreated by a real front projector.

An alternate application of the present technique is better understoodwith reference to FIGS. 23A to 23C. In the present example, immersiveprojector P2 of FIG. 23C will be used to create various ambient lightingeffects (i.e. virtual environments). If the camera is positioned suchthat its FOV covers a significant portion of the display room, one canuse view projection to create an immersive environment where the wallsare lit according to a virtual model. To achieve this, camera C istherefore positioned such that its FOV covers a significant portion of adisplay room 111, as shown in FIG. 23A. In FIG. 23A, camera C andimmersive projector P2 are positioned such that the FOV of camera Cencompasses most, if not all of (and preferably more than) the FOV ofimmersive projector P2. In the present example, P2 is shown as animmersive projector, but projector P2 may be any type of a projector,such a front projector. To establish a relationship between camera C andprojector P2, a light transport matrix T₃ relating camera C to projectorP2 is captured, i.e. determined, using any of the methods describedabove. Once this is done, an image c₃ as viewed (i.e. captured) bycamera C will be related to a projected image p₃, as projected byprojector P2, according to the following relationship:

c₃=T₃p₃

which results in

p ₃=(T ₃ ⁻¹)×(c ₃)

Consequently, one can build a virtual model of display surfaces of room111. This constructed virtual model room (i.e. virtual room) 111″, shownin FIG. 23B, may be a computer simulation, for example. Once virtualroom 111″ is created, various simulated lighting effects (or projectedimages or floating images) may be added to virtual room 111″. Forexample, FIG. 23B shows virtual room 111″ being lit by candle light froma large candle 113. The computer model further models the position andresolution of camera C (of FIG. 23A), shown as dotted box C in FIG. 23B.The computer model then “captures” (i.e. creates) a synthetic view c₃″of virtual room 111″ from the viewpoint camera C to simulate a realimage of virtual room 111″ as if it had been captured by real camera Cof FIG. 23A. The simulated lighting effects of FIG. 23B can then berecreated in real room 111 of FIG. 23C using P2 by projectingtransformed image (T₃ ⁻¹)×(c₃″).

An example of an application of this technique is shown in FIG. 24. Inthe present case, it is desired to project an image 117 that is biggerthan the walls of a real room 111. As was discussed above, varioustechniques may be used to calibrate a real projector to compensate forthe angles of the walls and ceiling to the projection wall of real room111, but the present invention solves this problem using a differentapproach. In the present example, virtual room 111″ (of FIG. 23B) hasdimensions similar to real room 111 (of FIGS. 23C and 24), and image 117is superimposed in an undistorted fashion onto virtual room 111″. A viewc₃″ (i.e. a synthetic captured image) of image 117 without distortion onvirtual room 111″ from the viewpoint of camera C is then created.Immersive projector P2 is then made to project transformed image (T₃⁻¹)×(c₃″) to recreate the undistorted oversized image 117 on a wall ofreal room 111. As is shown in FIG. 24, the result is an undistortedprojection that did not require calibrating projector P2 to compensatefor curvatures (or other irregularities) on a projection surface.

As seen, virtual projectors and environments can be combined to createan immersive movie viewer. Since the virtual environment is also anactive visual field, one can animate the larger field of view display tocreate a more engaging experience.

The above described techniques may be applied to the creation of largefield-of-view (i.e. large FOV) displays. A large FOV creates a sense ofimmersion and provides a more engaging experience for a viewer. Thepresent approach describes an immersive projection system with a verylarge FOV. The system is also designed with a built-in large FOVcamera/light sensor that is able to capture light from the areas coveredby projection's FOV. The sensor allows the system to adapt the projectedlight so as to optimize image quality and more generally allow thesystem to interact with its environment. Although the present system isprimarily motivated by the desire to display surround video content, itis important to note that this new projection system can also be used toview conventional video content.

With reference to FIG. 25, an exemplary projection system in accord withthe present invention in its minimal form consists of the followingcomponents: a projector 121; a camera 123, which can be a digital stillcamera or a digital video camera; curved mirror 125, which can bespherical or otherwise; and mounting mechanisms for the abovecomponents. Light from projector 121 is reflected off curved mirror 125before reaching a display surface 127, which can be any surface,including building walls, floors, ceilings, and dedicated projectionscreens. Display surface 127 can also be arbitrarily shaped. Reflectingthe projected light off the curved mirror enlarges the projector FOV.Light rays from the environment, which may or may not have originatedfrom the projector, also reflect off the curved mirror 125 beforereaching the camera. This similarly enlarges the camera FOV.

FIG. 26 shows a prototype based on the design of FIG. 25, and allelements in FIG. 26 similar to those of FIG. 25 have similar referencecharacters and are described above. The present construction highlightsone of the key applications of smart projector-camera systems, which isto build immersive multi-wall virtual environments. Thus, the presentexample uses a simple panoramic projection setup consisting of aconventional front projector 121, a high-resolution digital still camera123, and a hemispherical curved mirror 125. In the present setup, curvedmirror 125 (which may be termed a ‘dome projector’) is a low-costhemispherical plastic security mirror dome of the type used communityconvenience stores. This type of mirror dome costs at least three ordersof magnitude less than a professionally designed and fabricatedoptical-grade mirror. Furthermore, the mounting mechanism was also madefrom inexpensive parts available from typical hardware and buildingmaterial stores. As such, there is virtually no guarantee of conformanceto elegant mathematical models. In experimenting with this construct, itwas further found that mirror dome 125 is quite far from a truehemispherical surface (or any simple parametric form, for that matter).

FIG. 27 is an alternate view of the setup of FIG. 26, and shows the viewof mirror 125 as seen (very roughly) from the viewpoint of camera 123.As can be seen in the reflection of mirror 125 in FIG. 27, camera 123 isable to “see” the floor, at least three vertical walls, and the ceilingby means of the reflection in mirror 125.

In FIG. 28A a room with the present projection system installed is shownunder ambient lighting. FIG. 29A shows an uncalibrated dome projectordisplaying an image of a checkerboard. Clearly there is a significantamount of nonlinear geometric distortion in the displayed image. FIG.29B shows the same setup, but with geometric compensation using the viewprojection matrix. As the image was shot from the location of thereference camera, straight lines in the view remain straight acrossmultiple walls.

The view projection matrix {hacek over (T)}^(T) also compensatesphotometrically for the color and intensity variations as well asnon-uniformity in the display surface albedo/reflectance properties.FIG. 30A shows an uncalibrated dome projector displaying a uniformintensity image. As can been seen, the resulting image is significantlydarker towards the top left and right corners of the front wall. In FIG.30B, the same uniform intensity image is projected in a calibrated domeprojector, which is shown to produce a more uniform intensity.

Having a compact representation for all things necessary for viewprojection makes it easy to analyze display systems that could beoverwhelmingly difficult to calibrate otherwise. It also makes itpractical to build these systems and have them precisely calibrated upto the limits of the display and imaging hardware. In this section weillustrate the usefulness of T⁻¹ with a few examples using the domeprojector, described above, in combination with a front projector. Evenusing a poor man's panoramic projector like the one used in the presentexample, however, it was found that the view projection matrix stillenabled one to put together compelling, immersive display setups.

It is further to be understood that the present dome projector setup ofFIGS. 25-30 can be used in place of immersive projector P2 of FIGS. 18and 19 to achieve simulation of a front projector, as described above,or in place of immersive projector P2 in FIGS. 23A-23C to create virtualenvironments, as described above.

An example of immersive projection lighting created using the presentprojector-camera system is shown in FIG. 28B. As can be seen, thepresent projection-camera system is able to project images onto the twowalls as well as the ceiling. An example of how this effect is achievedusing the view projection matrix, {hacek over (T)}^(T), is illustratedin FIGS. 31-33.

The view projection matrix {hacek over (T)}^(T) is first generated usingany of the methods described above. As is explained above, when aprojection surface consists primarily of a flat surface (or conjoinedflat surfaces), forty-nine (49) or fewer, reference points may begenerated using seven vertical light lines and seven intersectinghorizontal light lines to approximate a full view projection matrix,{hacek over (T)}^(T), and still achieve a good level of projector tocamera calibration. In this case, the missing matrix entry values may beextrapolated from the 49 reference points since the projection surfaceis assumed to be flat.

However, since in the examples of FIGS. 25-30, the projection surfaceconsists of a curved mirror, it is preferred that a full view projectionmatrix {hacek over (T)}^(T) at the resolution of the projector begenerated. Since in the present example projector 121 has a resolutionof p×q projector pixels, calibration between projector 121 and camera123 should be achieved by generating p×q light transport referencepoints.

With reference to FIG. 31, projector 121 of FIG. 26 individuallyprojects a series of q vertical lines VL_1 to VL_q onto the projectionsurface (i.e. mirror 125 of FIG. 26 in the present case), which areindividually, and automatically, captured by camera 123 of FIG. 26. InFIG. 32, projector 121 then individually projects p horizontal linesHL_1 to HL_p that are in turn individually, and automatically, capturedby camera 123. As is explained above, the captured vertical andhorizontal lines are each individually combined to identify theiruniquely coincident reference point (i.e. light footprint). This processis continued until all unique intersecting points are identified (shownas white circles in FIG. 33), and their light transport informationextracted. It is to be understood that although the vertical andhorizontal lines emitted from projector 121 are perfectly vertical andhorizontal, the resultant projected lines on dome mirror 125 will followthe curvature of dome mirror 125.

As is explained above, it is preferred that the Display Constraint beenforced in the construction of view projection matrix {hacek over(T)}^(T). Thus, if one has a desired image, C, and wants to determinehow to transforme a projection image P in order to display desired imageC undistorted onto room 111 via mirror 125, one needs project adistorted version of image C defined as P={hacek over (T)}^(T)×C.

This is illustrated in FIG. 34, where Desired Image C is written as avector 200 consisting of m×n image pixel entry values, C₁ to C_(m×n).Vector 200 is multiplied with the created view projection matrix {hacekover (T)}^(T), which consists of (p×q) rows and (m×n) columns to producea (p×q) transformed image P, written as vector 201 and consisting of(p×q) image pixel entry values to be respectively applied theircorresponding one of the (p×q) projector pixels of projector 121. Theresultant transformed image P is shown to consist of p rows and qcolumns.

With reference to FIG. 35, transformed image P from FIG. 34 is sent toprojector 121 as Projector LCD Image P, and is projected onto mirror 125(FIG. 26). The resultant image 203 on room 111 is an undistortedrepresentation of Desired Image C of FIG. 35.

It is to be noted that in the present case, camera 123 and projector 121were not calibrated prior to creating transformed image P. Rather, thedistortion of transformed image P inherently compensates for issuesrelated to a lack of calibration between camera 123 and projector 121due to it having been constructed using view projection matrix {hacekover (T)}^(T), which includes calibration compensating information forthe camera-projector pair.

As it would be understood, if the desired image were a video image, thenthe view projection matrix {hacek over (T)}^(T) would be applied to thevideo image. That is, since a video image is comprised of a plurality ofstill images arranged in sequence, one would apply the view projectionmatrix {hacek over (T)}^(T) transformation to each of the sequencedstill images to produce a transformed video projection.

It should further be emphasized that the FOV of projector 121 and theFOV of camera 123 are in general different, and may or may not overlap.When there is a significant overlap in the two FOV's, images captured bycamera 123 can be used as feedback for improving the quality of aprojected image from projector 121 in a manner similar to thosedescribed above. For example, feedback from camera 123 to projector 121can be used to compensate for variations in a display surfacereflectance properties and shape (as seen by camera 123) so that aprojected image appears as though it were projected on a flat whitesurface.

The FOV of camera 123 may also include areas not covered by the FOV ofprojector 121. For example, while the FOV of projector 121 covers thefront and side walls of test room 127 shown in FIGS. 28A and 28B, thecamera may capture areas outside the projector's FOV, possibly includingareas where viewers are located. This allows the system to adapt andinteract with viewers either by detecting and tracking the viewers orthe viewers' pointing devices. It may be possible for camera 123 totrack small lights mounted, for example, on remote controls andfacilitate user interaction.

With reference to FIG. 36, an alternate configuration based on theconstruct of FIG. 25 but geared toward ceiling-mounted operation isshown. All elements similar to those of FIG. 25 have similar referencecharacters and are described above.

In FIGS. 37 and 38, two additional alternate configurations are shown.All elements in FIGS. 37 and 38 similar to those of FIG. 25 have similarreference characters and are described above.

In FIG. 37, a planar mirror 141 is used to fold the optical path so thatprojector 121 and camera 123 can be placed under the curved mirror 125,thereby achieving a smaller footprint. FIG. 38 shows a booth design forenclosing projector 121, camera 123, curved mirror 125, and flat mirror141 within a booth 143 for display booth operation. Using thisconstruct, one can simultaneously produce two projection images; a firstfront (or rear) projection image on a first Display Surface A and asecond rear projection image on a second Display Surface B.

In several of the designs described above, the projector and cameras donot have common optical centers. However it is possible to designprojector-camera pairs with collocated optical centers. While a systemwith collocated optical centers allows the projector and the camera tohave identical field-of-visions, a system with non-collocated opticalcenters has the potential to allow 3D reconstruction of its environment.

Up to this point, the provided examples have consisted of one camera andone projector, but as suggested above, multiple camera and projectorcombinations may be used. This poses the problem of how to seamlesslyintegrate, or combine, two or more projected images from two or moreprojectors that have different field-of-visions, or combine multipleprojectors to create a large field of vision display. Therefore, beforeexpounding on some of the visual effects possible by using multipleprojectors, it may be beneficial to first discuss how multipleprojectors may be seamlessly used together. That is, the efficacy of thepresent invention as applied to a single projector, single camerasystem, can be expanded to systems having multiple projectors and one(or more) cameras.

The development of a non-parametric method for calibratingprojector-camera systems and for solving the above described “viewprojection problem” is discussed above. That is, the view projectionproblem of defining how a projector should illuminate a scene (i.e.defining what a projector should project onto the scene) so that acamera sees a desired view is discussed above. In summary, theabove-described development first provided multiple constructs for alight transport matrix T that relates an image p projected from aprojector to an image c captured by a camera by the relationship c=Tp.Then, an interim working matrix, {hacek over (T)}, was defined as beingpopulated by the following relationship:

{hacek over (T)}r=Tr/(∥Tr∥)² , r=1, 2, 3, . . . , pq   (1)

where {hacek over (T)}r is the r^(th) column of {hacek over (T)} and pqis the number of columns in T. It was then shown that under the DisplayConstraint, one can define the view projection matrix {hacek over(T)}^(T) as:

{hacek over (T)} ^(T) =T ⁻¹

which leads to the following relation:

p={hacek over (T)}^(T) c

As is explained above, the Display Constraint comes from the observationthat in a typical projector-camera setup for information displaypurposes, any two distinct light rays emitting from distinct projectorpixels will typically hit a camera sensor pixel array at distinct parts,i.e., there is usually little overlap in the camera pixels i hit bylight from each of the distinct light rays. It implies that the columnsof T are orthogonal to each other, which enables the normalizationprocess in equation (1) to lead to the inverse of T. As is furtherexplained above, in situations where the Display Constraint is notobserved naturally, one can modify the construct of T to artificiallyimpose the Display Constraint by declaring the brightest pixel in eachrow of T as being part of a light footprint resulting from a distinctlight ray, and declaring all other pixels in the same row of T to bezero-valued entries. This operation forces T to become orthogonal, andthereby permits the application of equation (1).

The view projection matrix {hacek over (T)}^(T) thus solves the viewprojection problem, defined as: given a desired view c; find an imagep={hacek over (T)}^(T) c such that projecting p produces a scene which,when viewed from the location of the reference camera, has the sameappearance as c.

In practice, large field-of-view displays, virtual living stations withsurrounding wall screens, and many other applications require the use ofmore than one projector due to the limited field-of-view of eachindividual projector. In general when using more than one projector, thefield-of-views of these projectors partially overlap with each other. Toachieve view projection (or indeed to project a continuous image acrossthe different field-of-views) using display systems having two or moreprojectors, one needs to mosaic the individual projectors to generatethe desired camera view.

An example of mosaicing using the present invention is presented using atwo-projector display system. It is to be understood that the presentapproach to constructing a mosaic display may be extended to displaysystems having three or more projectors, since extension to systemsconsisting of more projectors is straightforward. That is, the processdescribed below for combining first projection image from a firstprojector with second projection image from a second projector can beapplied to combining the second projection image of the second projectorwith a third image of a third projector to create a mosaic imagecombining the first, second, and third projection images. Similarly, thesame process can be applied to combine the third projection image with afourth projection image from a fourth projector to create a mosaic imagethat combines the first, second, third, and fourth projection images.

Consider a multi-projector display system consisting of two projectorswith an overlapping field-of-view (FOV), and a camera with a FOV that isa superset of both projectors. The light transport matrixes T₁ and T₂(separately determined) respectively relating each projector image p₁and p₂ to corresponding camera captured image c₁ and c₂, give thefollowing equations:

c₁ =T₁p₁

and

c₂=T₂p₂

To display an image that spans the field-of-vision, FOV, of bothprojectors, one needs to find c₁ and c₂ such that a composite image, c,combines c₁ and c₂ seamlessly. That is, composite image c, which isdefined as c=c₁+c₂, is the desired image as observed by the camera. Morespecifically, one needs to compute the appropriate projector images p₁and p₂ to display the desired composite image c, by solving thefollowing linear equation,

c=c ₁ +c ₂

or

c=(T ₁ p ₁)+(T ₂ p ₂)=[T ₁ T ₂ ][p ₁ p ₂]^(T)   (2)

In such a setting, a camera pixel is lit by either one projector pixelfrom one of the projectors or two projector pixels simultaneously fromrespective projectors. In the former case, the camera pixel gives alinear equation on the corresponding projector pixel in p₁ or p₂. In thelatter case where the camera pixel falls in an overlapping part of theFOVs of the two projectors, it gives a linear equation on the twocorresponding projector pixels in p₁ or p₂, respectively. Since eachprojector pixel covers a number of camera pixels, it is constrained by anumber of linear equations. Thus such equations from all the camerapixels form an overconstrained linear system on the projector images p₁or p₂.

However, one can no longer directly compute the view projection matrixfor the two-projector system (i.e. one cannot compute the inverse of[T₁T₂] directly), as could be done in the single-projector cases,describe above. This is because the projection of pixels from differentprojectors can overlap with each other, and therefore the DisplayConstraint no longer holds between the multiple projectors, i.e. columnsin T₁ are not necessarily orthogonal to columns in T₂, and consequentlyEq. (1), above, can not be used to compute the view projection matrixfor [T₁T₂]. Therefore, instead of directly solving the linear system inEq. (2), an alternating linear solution has been developed by iteratingtwo steps until convergence, as follows:

-   -   1. Given the current estimate of p₂, compute p₁=({hacek over        (T)}₁)^(T)(c−T₂p₂)    -   2. Given the current estimate of p₁, compute p₂=({hacek over        (T)}₂)^(T)(c−T₁p₁)

In the two formulas immediately above, ({hacek over (T)}₁)^(T) and({hacek over (T)}₂)^(T) are the respective view projection matrices forthe two projectors, and p₂ is set equal to zero at the initial iterationstep. Since the view projection matrices naturally convey thecorrespondence and mapping between the images of the projectors and thecamera, it does not take many iterations for p₁ and p₂ in formulas 1 and2 (immediately above) to converge to their respective complementaryimage. In practice, the iteration process takes only a few iterationscycles (typically 5 or less) for p₁ and p₂ to converge to respectivecomplementing images that, when combined form a mosaic image. That is,p₁ will converge to a first image and p₂ will converge to a secondimage, and when images of p₁ and p₂ are projected and superimposed, theywill form a combined mosaic image.

With reference to FIG. 39, an example of a system that implements thisprocess includes a first projector-camera system pair 221 and a secondprojector-camera system pair 223. First projection-camera system pair221 includes a first projector 21 a and a first camera 25 a related by afirst view projection matrix {hacek over (T)}₁ constructed using anymethod described above. Similarly, second projection-camera system pair223 includes a second projector 21 b and a second camera 25 b related bya second view projection matrix T₂ also constructed using any of theabove-described methods. It is further to be understood that {hacek over(T)}₁ and {hacek over (T)}₂ are each generated independently such thatsecond projector-camera system pair 223 is off while {hacek over (T)}₁is generated and first projector-camera system pair 221 is off while{hacek over (T)}₂ is generated.

First projector-camera system pair 221 has a first field-of-vision FOV_1defining a first projection region Reg_1, and second projector-camerasystem pair 223 has a second field-of-vision FOV_2 defining a secondprojection region Reg_2. As shown, Reg_1 and Reg_2 overlap each otherwithin an area identified by crosshatch marks. This overlap region isfurther labeled Reg_1+2. It is to be understood that the size of overlapregion Reg_1+2 is made large for purposes of explanation and that asmall overlap region is more typical, although the amount of overlap isnot critical to the present application. p It is to be noted thatformulas p₁=({hacek over (T)}₁)^(T)(c−T₂p₂) and p₂=({hacek over(T)}₂)^(T)(c−T₁p₁) account for geometric (i.e. spatial) and photometric(i.e. lighting) compensation, and thus take care of any blending needsbetween adjoining displayed images. This is because light transportmatrix T incorporates geometric information, and consequently so doesview projection matrix {hacek over (T)}. Furthermore when identifyingthe light footprint information for a typical light transport matrix T,full photometric information is also obtained due to each captured pixelin a light footprint having a value-entry for any light intensityvariation of the three color (RGB) sub-components of each camera pixel(see for example, the white and shades of gray blocks that make up lightfootprints Ft1 in FIG. 3 a or Ft2 in FIG. 4A).

However, if matrix T were to be constructed in a binary ON/OFF manner(that is, pixels within a light footprint are classified as being fullyON (i.e. classified as having a light intensity value of 255 in atypical luminosity scale of 0 to 255) and pixels outside the lightfootprint are classified as being fully OFF (i.e. classified as having alight intensity value of 0), then this binary ON/OFF manner ofconstructing matrix T would not have much photometric information (sinceit would in effect be a black-and-white image). However, this binaryON/OFF manner of constructing matrix T would still have full geometricinformation so that the above two formulas (1) for p₁ and (2) for p₂would still be able to determine the geometric mosaicking of multipleprojection images. In such a case, however, an additional light blendingstep (described below) would be helpful to blend light intensities ofmultiple projection images when mosaicking the multiple projectionimages.

Another situation where such an additional step for blending the lightintensity of multiple projected images may be useful is in situationswhere light transport matrix T is created from a limited number of lightfootprint information. As is explained above, this would apply tosituations where a projection surface (or scene) is flat, and the lighttransport matrix T is estimated using a limited number of intersectingpatterns to generate a limited number of identified light footprints.Since the projection surface is flat, the missing geometric informationwithin light transport matrix T can be inferred using homographytechniques. The estimated light transport matrix T thus provides fullgeometric information, but it does not generally provide photometricinformation for locations between the limited number of identified lightfootprints.

If the projection surface is assumed to be uniformly white in color(such as a white projection canvas or a white wall), then a lightblending step may be skipped by simply insert white-color information asphotometric information for all identified and inferred light footprintinformation. Alternatively, if one assumes that the projection surfaceis of a uniform color, but not necessarily white, then one can define anestimated photometric reading by assigning it the photometricinformation value of one (or of an average of two or more) identifiedlight footprint(s). One can then populate the photometric information ofall inferred light footprints with the thus defined estimatedphotometric reading.

Returning now to the topic of the additional light blending step, it hasbeen found that even if one has a light footprint matrix T generated byusing all available pixel information (i.e. a full T, not an estimatedT), a projected image can still benefit from an additional lightblending step when mosaicking multiple projected images. In this case,the information obtained from the additional light blending step mayconstitute an additional parameter to the blending results obtained fromthe formulas p₁=({hacek over (T)}₁)^(T)(c−T₂p₂) and p₂=({hacek over(T)}₂)^(T)(c−T₁p₁).

A simplified view of multiple projection regions demonstrating thisadditional light blending step is shown in FIG. 40. As before, Reg_1 isthe projection region provided by projector-camera system pair 221, andReg_2 is the projection region provided by projector-camera system pair223. For ease of explanation, Reg_1 is denoted by vertical hatch linesand Reg_2 is denoted by horizontal hatch lines. Overlap region Reg_1+2is therefore denoted by the intersection of vertical and horizontalhatch lines. Of particular interest is the definition of a desiredprojection region 225 denoted by a darkened outline spanning acrossparts of regions Reg_1, Reg_2, and Reg_1+2. Desired projection region225 defines the region upon which a desired combined (i.e. mosaic) imageis to be displayed. Desired projection region 225 has imagecontributions from both projector-camera systems 221 and 223. Withindesired projection region 225, Reg_A identifies that part of imageregion 225 provided solely by projector-camera system 221, Reg_Cidentifies that part of image region 225 provided solely byprojector-camera system 223, and Reg_B identifies that part of imageregion 225 provided by a combination of projector-camera systems 221 and223. It is to be understood that Reg_B might also be provided solely byeither one of projector-camera systems 221 or 223, but it has been foundthat visual artifacts at an image border (where an image provided by oneprojector-camera system ends and a second image projected by a secondprojector-camera system begins) can be mitigated, or eliminated, byblending the transition between projector-camera systems. Therefore, inthe presently preferred embodiment, both projector-camera systems 221and 223 contribute to the image created within Reg_B. The question athand is, how much (i.e. what parts) of an image within Reg_B each ofprojector-camera systems 221 and 223 provides.

For ease of explanation, FIG. 41 shows Reg_1 and that part of desiredimage 225 within the FOV of projector-camera system 221, i.e. Reg_B. Thevertical hatch lines indicate that part of Reg_1 that is made dark dueto it not contributing to desired image 225. Arrows 1A, 1B, 1C, and 1Dindicate how normalized light intensity is varied as one moves away froma border of region Reg_1 toward Reg_B, and approaches a border ofdesired image 225 that is provided by projector-camera system 223. AreaReg_A is outside the combined section Reg_B, and is provided solely byprojector-camera system 221. As one traverses arrow 1A from the leftborder of Reg_1 and approaches Reg_B, arrow 1A is shown dark to indicatethat all image components are provided by projector-camera system 221.Following arrow 1A, as one enters Reg_B, arrow 1A is initially dark andis then lightened (i.e. shown as thinning stripes) to indicate that theimage intensity is initially strongly provided by projector-camerasystem 221, but the intensity drops off as one traverses from left toright along Array 1A toward the right border of Reg_B and Reg_1. Arrow1D indicates that initially at the right end of Reg_B, no intensity isprovided by projector-camera system 221, but the light intensity fromprojector-camera system 221 is increased as one traverses from right toleft within Reg_B toward the end-point of arrow 1A. Similarly, arrow 1Bindicates that the light intensity falls as one traverses down arrow 1Baway from a border of region 225. Likewise, arrow 1C indicates thatlight intensity falls as one traverses up arrow 1C away from a border ofregion 225.

In other words, light intensity variations in an image can be expressedas a factor of a defined maximum light intensity value, i.e. thenormalized value. This normalized valued multiplied by a factor of 1,would provide the defined maximum light intensity value, and the samenormalized value multiplied by a factor of 0.5 would provide half thedefined maximum light intensity value. By altering the normalized valuewithin a given region, one can alter the brightness of the image withinthe given region. In the present case, undesirable light artifacts areavoided at the borders by providing a gradual blending of image borders,rather than abruptly changing light intensity of pixels at the borders.

A similar construct is shown in FIG. 42 from the point of view ofprojection-camera system 223. Again, the horizontal hatch lines indicatethat part of Reg_2 that is made dark due to it not contributing todesired image 225. Also like before, Reg_B indicates the blending areawhere an image is provided by a combination of both projector-camerasystems 221 and 223, and region Reg_C indicates that part of desiredimage region 225 provided solely by projector-camera system 223. Arrows2A, 2B, 2C, and 2D are indicative of how the normalized light intensityis varied as moves from Reg_2 toward the borders of desired region 225,and within Reg_B toward that part of desired image 225 provided byprojection-camera system 221. Area Reg_C defines that part of desiredimage 225 that is outside combined region Reg_B, and is provided solelyby projector-camera system 223. As one enters region Reg_B, arrow 2Aindicates that the image intensity is initially strongly provided byprojector-camera system 223, but the intensity drops off as onetraverses from right to left along Array 2A toward the left border ofregion Reg_B. Arrow 2D indicates that initially at the left end ofregion Reg_B, no intensity is provided by projector-camera system 223,but the light intensity from projector-camera system 223 is increased asone traverses from left to right within Reg_B. Similarly, arrow 2Bindicates that the light intensity falls as one traverses down arrow 2Baway from a border of desired region 225 within Reg_B. Likewise, arrow2C indicates that light intensity falls as one traverses up arrow 2Caway from a border of region 225.

For each pixel projected by projector-camera system 221, a determinationis made of the projected pixel's proximity to the left, right, upper,and lower border of the Reg_1 and Reg_B. These parameters affect thenormalized light intensity of the projected pixel. The closer aprojected pixel from projector-camera system 221 is to any one border ofReg_1, the higher parameter contribution for that border, which makesfor a brighter normalized intensity. The same is true for a projectedpixel from projector-camera system 223 with determination of theprojected pixel's proximity to the four borders of Reg_2 and Reg_B.Additionally, the light intensity of projected pixels close to border isadjusted as one approaches the border so as to avoid abrupt lightintensity changes at the borders.

For example in FIG. 40, as one travels along arrow A1 in region Reg_Afrom left to right one will be moving from the left border of desiredimage region 225, which is provided wholly by projector-camera system221, to the left border of region Reg_2, and region Reg_B which isprovided by both projector-camera systems 221 and 223. As one movesalong arrow A1 from left to right, one is moving further away from theleft border of Reg_1, but the image is provided solely byprojector-camera system 221, and so the normalized light intensity ofpixels produced by projector-camera system 221 is at its normal, highestvalue. When one reaches the end of arrow A1 and reaches the beginning ofarrow A2, projector-camera system 221 is still at its highest normalizedvalue since light blending is just about to begin. At the start of arrowA2, projector-camera system 221 has its highest normalized lightintensity and projector camera system 223 has its lowest normalizedlight intensity. As one moves along arrow A2 from left to right (i.e.from Reg_A provided exclusively by projector-camera system 221 towardReg_C provide exclusively by projector-camera system 223), thenormalized light intensity of projector-camera system 221 is loweredfrom its highest to it lowest normalized light intensity value.Conversely, as one moves along arrow A2 from left to right, thenormalized light intensity of projector-camera system 223 is raised fromits lowest normalized light intensity value to its highest. It is to beunderstood that this light transition is not necessarily linear. Forexample, the greater changes in normalized light intensity of projectorcamera system 221 preferred occur as one approaches the right border orReg_B along arrow A2.

Therefore, at the end of arrow A2, projector-camera system 223 is at itshighest normalized light intensity, and projector-camera system 221 isat its lowest normalized light intensity. As one moves from left toright along arrow A3, projector-camera system 223 provides all theprojected light and no light contribution is provided byprojector-camera system 221. Thus, within region Rev_C, projector-camerasystem 223 may be maintained at its highest normalized light intensity.

The above formulas p₁=({hacek over (T)}₁)^(T)(c−T₂p₂) and p₂=({hacekover (T)}₂)^(T)(c−T₁p₁), using both geometric and photometricinformation, was tested by mosaicing two projectors on a statue, asshown in FIG. 43. This is not a contrived example, as the two projectorsare required to cover all surfaces visible from the camera. Using thissetup, a seamless, distortion-free (from the point of view of thecamera) image was successfully projected on the statue. One can alsoeasily perform relighting by view projecting images of the statue undervarying illumination conditions.

Construction of large FOV projector systems using the above method ofcombining multiple projectors is shown in FIGS. 44A and 44B. In FIG.44A, a single curved mirror 125 is used in combination with multipleprojector-camera pairs 145. In FIG. 44B, a single mirror pyramid 151 isused with multiple projector-camera pairs 145 to achieve a large FOV.With the construct of FIG. 44B, the optical centers of all projectorscan be collocated within the mirror pyramid, creating a single virtuallarge FOV projector. Similarly, the camera optical centers can also becollocated to create a single virtual large FOV camera.

FIG. 45 shows that multiple large FOV projectors 153 a and 153 b (suchas those shown in FIGS. 44A, 44B, or other large FOV projector systems)can be used to achieve an even larger overall projection FOV. One ormore conventional projectors 155 can also be used in combination. Asseen, the FOV of projector 153 a, as indicated by dotted line 157overlaps the FOV of projector 154 b, as indicated by dotted line 159, byan overlap amount 161. The images from projectors 153 a and 153 b may becombined using the method described above for combining multipleprojectors.

Another application of multiple projectors is in the field ofthree-dimensional imaging (i.e. 3D imaging, or perspective imaging),where an illusion of a 3D image is created. The creation of 3D imaginglends itself particularly well to the present use of light transferproperties in conjunction with digital projectors due to thecharacteristics of digital projectors, and in particular LCD projectors.

LCD projectors by nature emit polarized light. Thus if one has two LCDprojectors emitting polarized light of different orientations, a userwith a pair of polarized light glasses can see 3D visual content (i.e.3D images) since each eye of the user can see a different image, onefrom each projector.

One way to produce polarized light of different orientation is toposition the two projectors such that they are at 90 degrees to eachother. An example configuration is shown in FIG. 46, where a firstprojector 301 is oriented at an angle to second projector 303. Firstprojector 301 is preferably oriented ninety degree (90°) to secondprojector 303. A pair of polarized glasses 305 are also shown in FIG. 46for reference. This method of creating a 3D visual content does notrequire any modification to the projectors beyond simple stands, likestand 307 shown in FIG. 46.

A symbolic representation of the configuration of FIG. 46 is shown inFIGS. 47 and 48, where all elements similar to those of FIG. 46 areidentified by similar reference characters. Therefore in FIG. 47, secondprojector 303 is shown on its base in a horizontal direction, and firstprojector 301 is shown on its side, oriented vertically with referenceto second projector 303. Additionally shown for sake of completeness iscamera 301 a, which works in association with first projector 301 toform a projector-camera system, as described above, for purposes ofcalibrating first projector 301 or generating light transportinformation for first projector 301. Similarly for the sake ofcompleteness, camera 303 a is shown to form a projector-camera systemwith second projector 303, as described above, for purposes ofcalibrating second projector 303 or generating light transportinformation for second projector 303. It is to be understood that oncethe needed light transport and calibration information for first andsecond projectors 301 and 303 has been generated, cameras 301 a and 303a maybe removed as long as the remaining components of the illustratedsystem remain unmoved. Additionally, camera 301 a is shown as beingoriented on its side to reproduce the orientation of first projector301, but it is to be understood that camera 301 a may be used orientedwith its base horizontal to a floor without deviating from the presentinvention.

First projector 301 projects a first image 315 onto a first projectionsurface 317 (i.e. projection scene, such as a canvass or wall). In thispreferred embodiment, the projection pixels of first projector 301project colors using standard RGB format (Red, Green, Blue) usingrespective Red, Green, and Blue projection panels. As shown, it ispreferred that all RGB light from first projector 301 be polarized alonga common direction, as is symbolically represented by arrows R₁, G₁, andB₁ being oriented along a common direction parallel to the base of firstprojector 301. However, since first projector 301 is oriented on itsside, arrows R₁, G₁, and B₁ are shown vertical.

Second projector 303 projects a second image 311 onto a secondprojection surface 313. It is to be understood that first 301 and second302 projectors may project their respective image onto a commonprojection surface, as is required to create 3D visual content, asexplained below. For purpose of explanation, however, second image 311is shown on second projection surface 313 in order to highlight that theRed, Green, and Blue projection panels of second projector 303 arepreferably oriented in a common direction, and preferably orientedparallel to the base of second projector 303. Therefore, the projectedRGB light components from second projector 303 are symbolicallyillustrated as three horizontal arrows R₂, G₂, B₂.

In FIG. 48, polarized light glasses 305 are representatively shown ashaving a pair of polarized lenses 321 and 323. The polarization ofpolarized lenses 321 and 323 is illustratively shown as parallel linesoriented at different angles. That is, polarization of lens 321 is shownby vertical parallel lines, and polarization of lens 323 is shown ashorizontal parallel lines. In general, the polarization of one lens(323) to the other (321) should be similar to the polarization of oneprojector (303) to the other (301).

It is noted that some LCD projectors on the market may not have all theRGB panels transmitting polarized light at the same orientation. Forexample, the R panel may produce polarization in one orientation whilethe G,B panels produce polarization in a different orientation,typically at 90° to the R panel.

This is illustrated in the left-hand side of FIG. 49. On the left-handside of FIG. 49, projected second image 311 is illustratively shown tohave its G₂ and B₂ polarized light components in the horizontaldirection, but have its R₂ polarized light component orientedvertically, 90° to the G₂,B₂ components. Similarly, projected firstimage 315 is illustratively shown to have its G₁ and B₁ polarized lightcomponents in the vertical direction, but have its R₁ polarized lightcomponent oriented horizontally, 90° to the G₂,B₂ components. This, offcourse, would interfere with the creation of 3D visual content. Since itis desirable that the RGB light transmitted from each projector bepolarized light in the same orientation, this problem may be solved byswapping the R channel of the video signal going to the 2 projectors, asis illustrated in the right-hand side of FIG. 49. The swapping of Rchannel of the video signal is symbolically illustrated by dash arrowsindicating that channel R₁, which is part of the video signal that wouldhave been part of first image 315 before swapping, is now sent to theprojector producing second image 311. Similarly, the arrows show channelR₂ of the video image that would be projected as second image 311 beforeswapping, now being projected as part of first image 315. This swappingcan be done in hardware, software, or performed during image processing.

Thus, polarized light component R₂ from second projected image 311(which was previously sent to second projector 303) is instead sent to,and produced by, first projector 301 alongside G₁ and B₁. Similarly,polarized light component R₁ from first projected image 315 (which waspreviously sent to sent to first projector 301) is instead sent to, andproduced by, second projector 303 alongside G₂ and B₂.

Irrespectively of what steps are taken to produce two sets of polarizedimages, ImgA and ImgB, the result is as shown in FIG. 50, whereR_(A),G_(A),B_(A) and R_(B),G_(B),B_(B) represent two differentlypolarized images created by differently oriented projectors, such asshown in FIGS. 46, 47 and 49. 3D visual content may be created withinany region where the two differently polarized images, ImgA and ImgB,intersect. In the present case, this region is identified by a darkeneddashed perimeter 331, i.e. intersection region 331.

Previously, a method for capturing the light transport matrix using p+qinstead of p×q images was explained. The technique requires theconstruction of sets

Y={Y1, . . . , Yp}, and X={X1, . . . , Xq},

such that

∀i ∈ {1, . . . , (p×q)}, ∃ X _(j) , Y _(k) |Xj ∩Y _(k) ={i},

As is shown in FIG. 50, however, intersection region 331 is only asubset of both polarized images ImgA and ImgB. Since only a small numberof projector pixels S (defined as intersection region 331) is ofinterest, one only needs to capture the camera images C_(X), C_(Y) forthe subsets of X, Y corresponding to each element of S.

For professionally set up display environments, it is possible tocapture the light transport matrix in an environment free of ambientlight, i.e. the only light is emitted from the projectors. For casualhome setups, however, this is not a realistic requirement. One cantherefore also capture an image co when all the projectors in the systemare displaying an all-black image. This captured image is thus a measureof the amount of ambient light in the environment.

This allows one to create a new set of images D_(X),D_(Y) from C_(X),C_(Y) by subtracting c₀ from each element. In essence, one issubtracting the corresponding ambient light component from each capturedpixel of each captured image C_(X),C_(Y). If there is no ambient light,then one may use C_(X), C_(Y), directly.

The new sets of images D_(X),D_(Y) simulate images captured under noambient light, and can be used for feature detection. Assuming theexistence of ambient light, D_(X),D_(Y) are generated such that:

D _(X) ={d _(x) |d _(x) =c−c ₀ , c ∈ C _(X)}

D _(Y) ={d _(y) |d _(y) =c−c ₀ , c ∈ C _(Y)}

If one lets d_(xi), d_(yj) be the images corresponding to a projectorpixel k, then the corresponding column T_(k) in the light transportmatrix can be constructed as:

T _(k)=MIN(d _(xi) , d _(yj))

In a manner similar to that explained above.

Having generated light transport matrix T, or View Projection matrix{hacek over (T)}^(T), a composite image may be created by superimposingthe images from both projectors, as shown in FIG. 50, and using themosaicing techniques, as explained above. That is, the composite imagewithin intersection region 331 can be generated by iterating formulasp₁=({hacek over (T)}₁)^(T)(c−T₂p₂) and p₂=({hacek over(T)}₂)^(T)(c−T₁p₁). It is to be understood that in formulas p₁ and p₂,modified images generated from D_(X) and D_(Y) may be used in place of cif it is necessary to remove ambient light components.

An example of a 3D image, i.e. 3D visual content, produced withinintersection region 331 is illustrated in FIG. 51.

It should be added that the set of camera pixels lit by a projectorpixel k can be determined by thresholding the elements of T_(k) to get abinary vector B_(k). That is, once T_(k) is generated, its content canbe compared to a predefined light intensity threshold value (such as 25%of a maximum light intensity value), and only those components withinT_(k) greater than the threshold are deemed to part of an image. Othercomponents not greater than the threshold may be set to zero. Thecentroid of the camera image coordinates of the nonzero points in B_(k)then is an estimate of the camera coordinates corresponding to theprojector coordinate k.

In noisy settings, it is possible that not all the nonzero points inB_(k) are induced by light from the projector. One may therefore checkthat all the nonzero points in Bk are contained within a small rectangleof a preset size (i.e. rectangular area or other predefined regionspanning a predefined number of pixels). If this condition is notsatisfied, i.e. if no rectangle or region of predefined size is found,then the image B_(k) is discarded and detection is said to have failedon the image. This ensures that only high quality features are used insubsequent operations.

The size of the rectangular area, or other defined region shape, ispreferably made to have an area similar to (or substantially equal to)the area footprint created by a single projected pixel on a capturedcamera image under no noise conditions (i.e. no light noise conditions).For example, the defined rectangular region may be made to have an areajust big enough to encompass footprint ft2 in FIG. 4A. In this exampleof FIG. 4A, the defined rectangular region could be defined to be anarea defined by a cluster of 15 camera pixels on a captured image.

The defined area (of rectangular or of other shape) could also bedefined to be a predefined fraction of the camera's resolution. In otherwords, the area could be defined to be an area equivalent to a clusterof a predefined group of camera pixels, wherein the number of camerapixels in the group defined as a percentage of the pixel resolution ofthe camera used to capture the images used for calibrating the presentprojector system. This would be useful when the footprint sizes formultiple cameras of differing pixel resolution are previouslydetermined, and it is found that different cameras of similar resolutionhave similarly sized light footprints.

While the invention has been described in conjunction with severalspecific embodiments, it is evident to those skilled in the art thatmany further alternatives, modifications and variations will be apparentin light of the foregoing description. Thus, the invention describedherein is intended to embrace all such alternatives, modifications,applications and variations as may fall within the spirit and scope ofthe appended claims.

1. A method of creating a perspective image (3-Dimensional image or 3Dimage) as perceived from a predefined reference view point, comprising:providing a first digital projector having a first projection regionarranged along a first orientation with reference to a reference base;providing a second digital projector having a second projection regionarranged along a second orientation with reference to said referencebase, said second projection region at least partially overlapping saidfirst projection region; defining a 3D imaging region within theoverlapping area of said first projection region and said secondprojection region; establishing a first light transport matrix T₁covering the portion of the first projection region that encompassessaid 3D imaging region, said first light transport matrix T₁ relating afirst image c₁ as viewed from said predefined reference view point to afirst projected image p₁ from said first digital projector according torelation c₁=T₁p₁; establishing a second light transport matrix T₂covering the portion of the second projection region that encompassessaid 3D imaging region, said second light transport matrix T₂ relating asecond image c₁ as viewed from said predefined reference view point to asecond projected image p₂ from said second digital projector accordingto relation c₂=T₂p₂; providing a first imaging sequence to said firstdigital projector, said first imaging sequence providing a first angledview of an image subject within said 3D imaging region with reference tosaid reference view point as defined by said first light transportmatrix T₁; providing a second imaging sequence to said second digitalprojector, said second imaging sequence providing a second angled viewof said image subject within said 3D imaging region with reference tosaid reference view point as defined by said second light transportmatrix T₂.
 2. The method of claim 1, wherein said first light transportmatrix is T₁ is constructed only for projector pixels of said firstdigital projector that contribute to said 3D imaging region.
 3. Themethod of claim 1, further providing a pair of eyeglasses at saidpredefined reference view point, said pair of eyeglasses having: a firstpolarized lens for capturing light from said first digital projector andrejecting light from said second digital projector; and a secondpolarized lens for capturing light from said second digital projectorand rejecting light from said first second digital projector.
 4. Themethod of claim 1, wherein said first orientation is at 90 degrees tosaid second orientation.
 5. The method of claim 1, wherein: said firstdigital projector has a first set of Red, Green, and Blue projectionpanels, two projection panels within said first set being polarizedalong a first polarization orientation with reference to said referencebase, and a third projection panel within said first set being polarizedalong a second polarization orientation with reference to said referencebase; said second digital projector has a second set of Red, Green, andBlue projection panels, two projection panels within said second setbeing polarized along said second polarization orientation withreference to said reference base, and a third projection panel withinsaid second set being polarized along said first polarizationorientation with reference to said reference base; identifying firstimaging information within said first imaging sequence corresponding tosaid third projection panel within said first set, and transferring tosaid second digital projector said first imaging information forprojection by said second digital projector; identifying second imaginginformation within said second imaging sequence corresponding to saidthird projection panel within said second set, and transferring to saidfirst digital projector said second imaging information for projectionby said first digital projector.
 6. The method of claim 1, wherein: saidfirst imaging sequence has R1 (Red 1), G1 (Green 1), and B1 (Blue 1)imaging channels; said second imaging sequence has R2 (Red 2), G2 (Green2), and B2 (Blue 2) imaging channels; said method further comprising,swapping the R1 and R2 channels of said first and second imagingsequences.
 7. The method of claim 1, wherein establishing said firstlight transport matrix T₁, includes: using a camera at said predefinedreference view point, capturing a reference noise image C₀ of at leastpart of said first projection region under ambient light with no imageprojected from said first digital projector; using said camera at saidpredefined reference view point, capturing a first image of a first testpattern projected by said first digital projector within said firstprojection region, said first test pattern being created bysimultaneously activating a first group of projection pixels within saidfirst digital projector, all projection pixels not in said first testpattern being turned off; using said camera at said predefined referenceview point, capturing a second image of a second test pattern projectedby said first digital projector within said first projection region,said second test pattern being created by simultaneously activating asecond group of projection pixels within said first digital projector,all projection pixels not in said second test pattern being turned off,wherein said first and second groups of projection pixels have only oneprojection pixel in common defining a target projection pixel;subtracting said reference noise image C₀ from said first image to makea first corrected image; subtracting said reference noise image C₀ fromsaid second image to make a second corrected image; comparing imagepixels of said first corrected image to corresponding image pixels ofsaid second corrected image and retaining the darker of two comparedimage pixels, the retained image pixels constituting a composite image;identifying all none-dark image pixels in said composite image;selectively identifying said none-dark pixels in said composite image asnon-zero light transport coefficients associated with said targetprojection pixel.
 8. The method of claim 7, wherein only pixels withinsaid composite image having a light intensity value not less than apredefined minimum value are identified as said none-dark image pixels.9. The method of claim 8, wherein said predefined minimum value is 25%of the maximum light intensity value.
 10. The method of claim 7, whereinidentified non-dark image pixels are arranged into adjoining pixelcluster according to their pixel position relation within an image, andonly pixels within a cluster having a size not less than a predefinednumber of pixels are identified as non-dark pixels.
 11. The method ofclaim 10, wherein said predefined number of pixels is defined by thenumber of pixels within an average sized light footprint created on thecamera-pixel-array within said camera created by turning ON a singleprojection pixel within said first digital projector.
 12. The method ofclaim 10, wherein the centroid within a cluster having a size not lessthan said predefined number of pixels constitutes an estimate of thecamera pixel array coordinate corresponding to said target projectionpixel.
 13. The method of claim 1, further comprising: (a) defining adesired composite image “c”; (b) setting p₂ equal to zero, solving forp₁ in formula p₁=T₁ ⁻¹(c−T₂p₂); (c) using the current solved value forp₁, solving for p₂ in formula p₂=T₂ ⁻¹(c−T₁p₁); (d) using the currentsolved value for p₂, solving for p₁ in formula p₁=T₁ ⁻¹(c−T₂p₂); andrepeating steps (c) and (d) in sequence until p₁ converges to a firstmosaicing image and p₂ converges to a second mosaicing image.
 14. Themethod of claim 13, wherein said desired composite image “c” is definedas “c”=c₁+c₂.
 15. The method of claim 13, wherein at least one of T₁ ⁻¹or T₂ ⁻¹ is an estimate of an inverse of a general light transportmatrix T, said estimate being generated by: identifying in turn, eachcolumn in said general light transport matrix T as a target column,calculating normalized values for not-nullified entry values in saidtarget column with reference to said target column; creating anintermediate matrix of equal size as said general light transport matrixT; populating each column in said intermediate matrix with thecalculated normalized values of its corresponding target column in saidgeneral light transport matrix T, each normalized value in eachpopulated column in said intermediate matrix maintaining a one-to-onecorrespondence with said not-nullified entry values in its correspondingcolumn in said general light transport matrix T; applying a transposematrix operation on said intermediate matrix.
 16. The method of claim15, wherein if said intermediate matrix is denoted as {hacek over (T)},a target column in general light transport matrix T is denoted as Tr anda corresponding column in {hacek over (T)} is denoted as {hacek over(T)}r, then the construction and population of {hacek over (T)} isdefined as {hacek over (T)}r=Tr/(∥Tr∥)² such that p₁=({hacek over(T)}₁)^(T)(c−T₂p₂) if T₁ ⁻¹ is the estimate of the inverse of generallight transport matrix T⁻¹ and p₂=(T₂)^(T)(c−T₁p₁) if T₂ ⁻¹ is theestimate of the inverse of a general light transport matrix T⁻¹.
 17. Themethod of claim 13, wherein at least one of T₁ or T₂ is an estimatedlight transport matrix generated from a sampling of light footprintinformation not having a one-on-one correspondence with every projectorpixel in its respective digital projector.